Provide options to customize Katana's behavior when indexing custom endpoints from the Developer Docs section. This could involve allowing a custom `config.katana.txt` or `config.katana.flags` file, or a per-URL configuration for Katana.
Hi, I think it would be pretty helpful to be able to provide some customization to the katana behavior when indexing a custom endpoint provided in the Developer Docs section. The best solution would probably be to have a per-URL configuration that allows changing the katana behavior but looking at the current implementation I think an easy "fix" would be to allow specifying a custom prefix URL and, globally, allow creating an optional `config.katana.txt` or `config.katana.flags` alongside the config.toml file that gets loaded using katana's `--config` flag if detected. This would allow full customization of the katana behavior without increasing complexity of the current `WebCrawlerJob` implementation. **Additional context** See [katana config flag](https://github.com/projectdiscovery/katana/blob/6253b22ac5a90a7c5e2af870cf0c551cbc98c69d/cmd/katana/main.go#L130) and current [`crawl_url` implementation](https://github.com/TabbyML/tabby/blob/aa5d0e4cb73b77d74923c140a5030fd8ce32b34b/cra