summaryrefslogtreecommitdiffstats
path: root/hugolib/sitemap_test.go
AgeCommit message (Collapse)Author
2024-03-16hugolib: Fix sitemap index with monolingual siteJoe Mooring
Fixes #12266
2024-03-01Make sure that sitemaps gets generated even if there is a content bundle ↵Bjørn Erik Pedersen
with the same path Fixes #12183
2024-02-21Sitemap should not use list.xmlBjørn Erik Pedersen
Fixes #12101
2024-01-28testing: Simplify some integration testsBjørn Erik Pedersen
2024-01-27all: Rework page store, add a dynacache, improve partial rebuilds, and some ↵develop2024Bjørn Erik Pedersen
general spring cleaning There are some breaking changes in this commit, see #11455. Closes #11455 Closes #11549 This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build. The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate. A list of the notable new features: * A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server. * A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently. * You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs. We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections). Memory Limit * Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory. New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively. This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive): Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers. Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds. Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role). Fixes #10169 Fixes #10364 Fixes #10482 Fixes #10630 Fixes #10656 Fixes #10694 Fixes #10918 Fixes #11262 Fixes #11439 Fixes #11453 Fixes #11457 Fixes #11466 Fixes #11540 Fixes #11551 Fixes #11556 Fixes #11654 Fixes #11661 Fixes #11663 Fixes #11664 Fixes #11669 Fixes #11671 Fixes #11807 Fixes #11808 Fixes #11809 Fixes #11815 Fixes #11840 Fixes #11853 Fixes #11860 Fixes #11883 Fixes #11904 Fixes #7388 Fixes #7425 Fixes #7436 Fixes #7544 Fixes #7882 Fixes #7960 Fixes #8255 Fixes #8307 Fixes #8863 Fixes #8927 Fixes #9192 Fixes #9324
2023-07-08Fix defaultContentLanguageInSubdir with only 1 languageBjørn Erik Pedersen
Fixes #10064
2023-05-16Create a struct with all of Hugo's config optionsBjørn Erik Pedersen
Primary motivation is documentation, but it will also hopefully simplify the code. Also, * Lower case the default output format names; this is in line with the custom ones (map keys) and how it's treated all the places. This avoids doing `stringds.EqualFold` everywhere. Closes #10896 Closes #10620
2022-04-08Rework the Destination filesystem to make --renderStaticToDisk workBjørn Erik Pedersen
See #9626
2022-03-17all: gofmt -w -r 'interface{} -> any' .Bjørn Erik Pedersen
Updates #9687
2021-02-06tpl/embedded: Exclude pages without Permalink from sitemapJeremy Epstein
2020-12-03all: Format code with gofumptBjørn Erik Pedersen
See https://github.com/mvdan/gofumpt
2019-12-12Rework template handling for function and map lookupsBjørn Erik Pedersen
This is a big commit, but it deletes lots of code and simplifies a lot. * Resolving the template funcs at execution time means we don't have to create template clones per site * Having a custom map resolver means that we can remove the AST lower case transformation for the special lower case Params map Not only is the above easier to reason about, it's also faster, especially if you have more than one language, as in the benchmark below: ``` name old time/op new time/op delta SiteNew/Deep_content_tree-16 53.7ms ± 0% 48.1ms ± 2% -10.38% (p=0.029 n=4+4) name old alloc/op new alloc/op delta SiteNew/Deep_content_tree-16 41.0MB ± 0% 36.8MB ± 0% -10.26% (p=0.029 n=4+4) name old allocs/op new allocs/op delta SiteNew/Deep_content_tree-16 481k ± 0% 410k ± 0% -14.66% (p=0.029 n=4+4) ``` This should be even better if you also have lots of templates. Closes #6594
2019-08-12tests: Convert from testify to quicktestBjørn Erik Pedersen
2019-05-02output: Fix permalink in sitemap etc. when multiple permalinkable output formatsBjørn Erik Pedersen
In Hugo 0.55.0 we made AMP `permalinkable`. We also render the output formats in their natural sort order, meaning `AMP` will be rendered before `HTML`. References in the sitemap would then point to the AMP version, and this is normally not what you'd want. This commit fixes that by making `HTML` by default sort before the others. If this is not you want, you can set `weight` on the output format configuration. Fixes #5910
2019-03-23Make Page an interfaceBjørn Erik Pedersen
The main motivation of this commit is to add a `page.Page` interface to replace the very file-oriented `hugolib.Page` struct. This is all a preparation step for issue #5074, "pages from other data sources". But this also fixes a set of annoying limitations, especially related to custom output formats, and shortcodes. Most notable changes: * The inner content of shortcodes using the `{{%` as the outer-most delimiter will now be sent to the content renderer, e.g. Blackfriday. This means that any markdown will partake in the global ToC and footnote context etc. * The Custom Output formats are now "fully virtualized". This removes many of the current limitations. * The taxonomy list type now has a reference to the `Page` object. This improves the taxonomy template `.Title` situation and make common template constructs much simpler. See #5074 Fixes #5763 Fixes #5758 Fixes #5090 Fixes #5204 Fixes #4695 Fixes #5607 Fixes #5707 Fixes #5719 Fixes #3113 Fixes #5706 Fixes #5767 Fixes #5723 Fixes #5769 Fixes #5770 Fixes #5771 Fixes #5759 Fixes #5776 Fixes #5777 Fixes #5778
2017-06-13all: Update import paths to gohugoio/hugoBjørn Erik Pedersen
2017-06-06hugolib: Add test for no 404 in sitemapBjørn Erik Pedersen
Closes #3563
2017-04-02tpl: Rework to handle both text and HTML templatesBjørn Erik Pedersen
Before this commit, Hugo used `html/template` for all Go templates. While this is a fine choice for HTML and maybe also RSS feeds, it is painful for plain text formats such as CSV, JSON etc. This commit fixes that by using the `IsPlainText` attribute on the output format to decide what to use. A couple of notes: * The above requires a nonambiguous template name to type mapping. I.e. `/layouts/_default/list.json` will only work if there is only one JSON output format, `/layouts/_default/list.mytype.json` will always work. * Ambiguous types will fall back to HTML. * Partials inherits the text vs HTML identificator of the container template. This also means that plain text templates can only include plain text partials. * Shortcode templates are, by definition, currently HTML templates only. Fixes #3221
2017-04-02Revert "tpl: Rework to handle both text and HTML templates"Bjørn Erik Pedersen
Will have to take another stab at this ... This reverts commit 5c5efa03d2512749950b0d05a7d4bde35ecbdc37. Closes #3260
2017-04-02tpl: Rework to handle both text and HTML templatesBjørn Erik Pedersen
Before this commit, Hugo used `html/template` for all Go templates. While this is a fine choice for HTML and maybe also RSS feeds, it is painful for plain text formats such as CSV, JSON etc. This commit fixes that by using the `IsPlainText` attribute on the output format to decide what to use. A couple of notes: * The above requires a nonambiguous template name to type mapping. I.e. `/layouts/_default/list.json` will only work if there is only one JSON output format, `/layouts/_default/list.mytype.json` will always work. * Ambiguous types will fall back to HTML. * Partials inherits the text vs HTML identificator of the container template. This also means that plain text templates can only include plain text partials. * Shortcode templates are, by definition, currently HTML templates only. Fixes #3221
2017-02-18hugolib: More test helper cleanupBjørn Erik Pedersen
2017-02-17hugolib: Test helper cleanupBjørn Erik Pedersen
2017-02-17tpl: Refactor packageBjørn Erik Pedersen
Now: * The template API lives in /tpl * The rest lives in /tpl/tplimpl This is bound te be more improved in the future. Updates #2701
2017-02-17all: Refactor to nonglobal Viper, i18n etc.Bjørn Erik Pedersen
This is a final rewrite that removes all the global state in Hugo, which also enables the use if `t.Parallel` in tests. Updates #2701 Fixes #3016
2017-02-04all: Refactor to nonglobal file systemsBjørn Erik Pedersen
Updates #2701 Fixes #2951
2017-01-07all: Refactor to non-global loggerBjørn Erik Pedersen
Note that this looks like overkill for just the logger, and that is correct, but this will make sense once we start with the template handling etc. Updates #2701
2017-01-03Update documentation and test for sitemap templateDaniel Lang
2016-11-23hugolib: Camelcase ALL_CAPS vars and const in testsbogem
See #2014
2016-11-22node to page: Add all nodes to sitemapBjørn Erik Pedersen
Updates #2297 Fixes #1303
2016-10-24all: Unify case of config variable namesAlbert Nigmatzianov
All config variables starts with low-case and uses camelCase. If there is abbreviation at the beginning of the name, the whole abbreviation will be written in low-case. If there is abbreviation at the end of the name, the whole abbreviation will be written in upper-case. For example, rssURI.
2016-09-06Render main content language in root by defaultBjørn Erik Pedersen
Fixes #2312
2016-09-06Improve language handling in URLsBjørn Erik Pedersen
The current "rendering language" is needed outside of Site. This commit moves the Language type to the helpers package, and then used to get correct correct language configuration in the markdownify template func. This commit also adds two new template funcs: relLangURL and absLangURL. See #2309
2016-09-06Optimize the multilanguage build processBjørn Erik Pedersen
Work In Progress! This commit makes a rework of the build and rebuild process to better suit a multi-site setup. This also includes a complete overhaul of the site tests. Previous these were a messy mix that were testing just small parts of the build chain, some of it testing code-paths not even used in "real life". Now all tests that depends on a built site follows the same and real production code path. See #2309 Closes #2211 Closes #477 Closes #1744
2016-09-06Introduce HugoSites typeBjørn Erik Pedersen
And a Hugo global variable which contains the site under build. This is really needed to get some level of control of the "multiple languages" in play. There are still work related to this scattered around, but that will come. With this commit, the multilingual feature is starting to work.
2016-09-06Add multilingual support in HugoAlexandre Bourget
Implements: * support to render: * content/post/whatever.en.md to /en/2015/12/22/whatever/index.html * content/post/whatever.fr.md to /fr/2015/12/22/whatever/index.html * gets enabled when `Multilingual:` is specified in config. * support having language switchers in templates, that know where the translated page is (with .Page.Translations) (when you're on /en/about/, you can have a "Francais" link pointing to /fr/a-propos/) * all translations are in the `.Page.Translations` map, including the current one. * easily tweak themes to support Multilingual mode * renders in a single swift, no need for two config files. Adds a couple of variables useful for multilingual sites Adds documentation (content/multilingual.md) Added language prefixing for all URL generation/permalinking see in the code base. Implements i18n. Leverages the great github.com/nicksnyder/go-i18n lib.. thanks Nick. * Adds "i18n" and "T" template functions..
2016-04-08hugolib: Un-export internal Site-methodsBjørn Erik Pedersen
These are obviously internal and for the most part undocumented, creating lots of GoLint warnings. See #1160 See #2014
2016-04-07hugolib: Camelcase ALL_CAPS vars and const in testsBjørn Erik Pedersen
See #2014
2016-03-31Add readFile template funcBjørn Erik Pedersen
This also includes a refactor of the hugofs package and its usage. The motivation for that is: The Afero filesystems are brilliant. Hugo's way of adding a dozen of global variables for the different filesystems was a mistake. In readFile (and also in some other places in Hugo today) we need a way to restrict the access inside the working dir. We could use ioutil.ReadFile and implement the path checking, checking the base path and the dots ("..") etc. But it is obviously better to use an Afero BasePathFs combined witha ReadOnlyFs. We could create a use-once-filesystem and handle the initialization ourselves, but since this is also useful to others and the initialization depends on some other global state (which would mean to create a new file system on every invocation), we might as well do it properly and encapsulate the predefined set of filesystems. This change also leads the way, if needed, to encapsulate the file systems in a struct, making it possible to have several file system sets in action at once (parallel multilanguage site building? With Moore's law and all...) Fixes #1551
2016-03-09Create template clone for late template executionBjørn Erik Pedersen
Fixing some breaking blogs on Go 1.6 Fixes #1879
2016-02-07hugolib: Add test for parseSitemapBjørn Erik Pedersen
2016-01-05Add embeded template for robots.txtIvan Fraixedes
2015-12-10Add copyright header to that source files that don’t have one.Anthony Fok
See #1646
2015-05-20Stop Viper from leaking across many of the tests (now tests pass regardless ↵spf13
of order tested)
2015-03-18Change safeHTtml to safeHTML in sitemap templatebep
2014-11-04Updating tests to use new Targets & Writers and switch to using ↵spf13
Afero.MemMapFs for more accurate tests.
2014-05-09Add Sitemap testingVincent Batoufflet