User-agent: * Disallow: /App-Code/ Disallow: /App-Data/ Disallow: /Bin/ Disallow: /RPDataExtraction/ Disallow: /en/webinar/ Disallow: /catalog.wci Allow: /*?$ #allow files that contain question mark somewhere in the url string