feat(crawlers)!: Replace HTTP client persist_cookies_per_session with crawler save_response_cookies

docs/03_concepts/15_cookie_management.mdx

@@ -17,7 +17,7 @@ import PersistCookies from '!!raw-loader!roa-loader!./code_examples/cookie_manag
 
 Cookies carry login state, consent flags, CSRF tokens, and other per-user data that a site expects to see on every request.
 
-In Crawlee, cookies are stored on the <ApiLink to="class/Session">`Session`</ApiLink> that handles a request, in a <ApiLink to="class/SessionCookies">`SessionCookies`</ApiLink> jar. When a request runs on a session, the HTTP client sends the session's cookies with the request and writes any `Set-Cookie` from the response back onto the same jar. The next request on that session carries the updated cookies.
+In Crawlee, cookies are stored on the <ApiLink to="class/Session">`Session`</ApiLink> that handles a request, in a <ApiLink to="class/SessionCookies">`SessionCookies`</ApiLink> jar. When a request runs on a session, the HTTP client sends the session's cookies with the request and writes any `Set-Cookie` from the response back onto the same jar. The next request on that session carries the updated cookies. To keep the response cookies out of the session, set <ApiLink to="class/HttpCrawlerOptions#save_response_cookies">`save_response_cookies`</ApiLink> to `False` on the crawler. The session's cookies are still sent.
 
 Cookies follow the session, not the URL. Two requests share cookies when they run on the same session. For how sessions are created and rotated, see the [Session management](./session-management) guide.
 
@@ -49,7 +49,7 @@ To start every session with a known set of cookies, for example a consent flag o
 
 ## Cookies with PlaywrightCrawler
 
-<ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> keeps the session jar and the browser context in sync. Before navigation, the session's cookies are loaded into the browser context. After the handler returns, cookies from the browser context are merged back onto the session. The merge captures whatever the page picked up through `Set-Cookie`, JavaScript, or redirects.
+<ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> keeps the session jar and the browser context in sync. Before navigation, the session's cookies are loaded into the browser context. After the handler returns, cookies from the browser context are merged back onto the session. The merge captures whatever the page picked up through `Set-Cookie`, JavaScript, or redirects. With <ApiLink to="class/PlaywrightCrawler#__init__">`save_response_cookies`</ApiLink> set to `False`, the crawler skips the merge.
 
 The first request receives a cookie in the browser. Crawlee merges it onto the session, so a later request on the same session sees it.
 

docs/06_upgrading/upgrading_to_v2.md

@@ -389,7 +389,21 @@ A `Cookie` header passed with a request or set on the client is merged with the
 
 `send_request` and `context.send_request` take a new `cookies` argument, the cookies to send and store instead of the session cookies for a single call. To send no session cookies, pass `cookies=SessionCookies()`. With `PlaywrightCrawler`, the response cookies still reach the session through the browser context.
 
-Without a session, or with `persist_cookies_per_session=False`, the cookies a server sets during a redirect chain are sent with the following requests of that chain. In v1, `ImpitHttpClient` and `HttpxHttpClient` didn't send them.
+Without a session, or with `save_response_cookies=False` on the crawler, the cookies a server sets during a redirect chain are sent with the following requests of that chain. In v1, `ImpitHttpClient` and `HttpxHttpClient` didn't send them.
+
+### `save_response_cookies` replaces `persist_cookies_per_session`
+
+`persist_cookies_per_session` is removed from the HTTP clients. To stop saving the response cookies to the session, pass `save_response_cookies=False` to the crawler. The session cookies are still sent. `PlaywrightCrawler` and `StagehandCrawler` take the same option, which stops saving the cookies of the page to the session after the request handler. In `AdaptivePlaywrightCrawler`, the option applies to both the HTTP and the browser crawling.
+
+```python
+# v1
+crawler = ParselCrawler(http_client=ImpitHttpClient(persist_cookies_per_session=False))
+
+# v2
+crawler = ParselCrawler(save_response_cookies=False)
+```
+
+With `BasicCrawler` and the HTTP crawlers, `context.send_request` always saves the response cookies to the session, unless you pass other cookies in its `cookies` argument. In v1, `persist_cookies_per_session=False` applied to `context.send_request` too. With `PlaywrightCrawler` and its default HTTP client, `context.send_request` stores the response cookies in the browser context, so `save_response_cookies=False` keeps them out of the session as well.
 
 ### Some client arguments are ignored
 

src/crawlee/_types.py

@@ -593,7 +593,7 @@ def __call__(
             payload: The payload to include in the request.
             cookies: The cookies to send and store instead of the cookies of the session. With `PlaywrightCrawler`,
                 the response cookies are stored in the browser context as well, and so in the session after the
-                request handler.
+                request handler unless `save_response_cookies` is disabled.
             timeout: Maximum time allowed to receive the response, including all redirects, and to read its body.
                 Without it, the default timeout of the HTTP client applies.
 

src/crawlee/crawlers/_abstract_http/_abstract_http_crawler.py

@@ -16,6 +16,7 @@
 from crawlee._utils.urls import convert_to_absolute_url, is_url_absolute, to_absolute_url_iterator
 from crawlee.crawlers._basic import BasicCrawler, BasicCrawlerOptions, ContextPipeline
 from crawlee.errors import RequestThrottledError, SessionError
+from crawlee.sessions import SessionCookies
 from crawlee.statistics import StatisticsState
 
 from ._http_crawling_context import HttpCrawlingContext, ParsedHttpCrawlingContext, TParseResult, TSelectResult
@@ -45,6 +46,11 @@ class HttpCrawlerOptions(
     navigation_timeout: NotRequired[timedelta | None]
     """Timeout for the HTTP request."""
 
+    save_response_cookies: NotRequired[bool]
+    """Whether to save the cookies set by the responses to the session. The session cookies are sent either way. The
+    option doesn't apply to `context.send_request`.
+    """
+
 
 @docs_group('Crawlers')
 class AbstractHttpCrawler(
@@ -71,10 +77,12 @@ def __init__(
         *,
         parser: AbstractHttpParser[TParseResult, TSelectResult],
         navigation_timeout: timedelta | None = None,
+        save_response_cookies: bool = True,
         **kwargs: Unpack[BasicCrawlerOptions[TCrawlingContext, StatisticsState]],
     ) -> None:
         self._parser = parser
         self._navigation_timeout = navigation_timeout or timedelta(minutes=1)
+        self._save_response_cookies = save_response_cookies
         self._pre_navigation_hooks: list[Callable[[BasicCrawlingContext], Awaitable[None]]] = []
         self._post_navigation_hooks: list[Callable[[HttpCrawlingContext], Awaitable[None]]] = []
         self._shared_navigation_timeouts: dict[int, SharedTimeout] = {}
@@ -104,7 +112,7 @@ class _ParsedHttpCrawler(AbstractHttpCrawler):
             def __init__(
                 self,
                 parser: AbstractHttpParser[TParseResult, TSelectResult] = static_parser,  # ty: ignore[invalid-parameter-default]
-                **kwargs: Unpack[BasicCrawlerOptions[ParsedHttpCrawlingContext[TParseResult]]],
+                **kwargs: Unpack[HttpCrawlerOptions[ParsedHttpCrawlingContext[TParseResult]]],
             ) -> None:
                 kwargs['_context_pipeline'] = self._create_static_content_crawler_pipeline()
                 super().__init__(
@@ -297,13 +305,19 @@ async def _make_http_request(self, context: BasicCrawlingContext) -> AsyncGenera
         Yields:
             The original crawling context enhanced by HTTP response.
         """
+        # Without saving, the session cookies are still sent, but the response cookies go to a copy.
+        cookies = None
+        if context.session is not None and not self._save_response_cookies:
+            cookies = SessionCookies(context.session.cookies)
+
         async with self._shared_navigation_timeouts[id(context.request)] as remaining_timeout:
             response = await self._http_client.send_request(
                 context.request.url,
                 method=context.request.method,
                 headers=context.request.headers,
                 payload=context.request.payload,
                 session=context.session,
+                cookies=cookies,
                 proxy_info=context.proxy_info,
                 timeout=remaining_timeout,
             )

src/crawlee/crawlers/_adaptive_playwright/_adaptive_playwright_crawler.py

@@ -125,6 +125,7 @@ def __init__(
         result_checker: Callable[[RequestHandlerRunResult], bool] | None = None,
         result_comparator: Callable[[RequestHandlerRunResult, RequestHandlerRunResult], bool] | None = None,
         playwright_crawler_specific_kwargs: _PlaywrightCrawlerAdditionalOptions | None = None,
+        save_response_cookies: bool = True,
         statistics: Statistics[AdaptivePlaywrightCrawlerStatisticState] | None = None,
         **kwargs: Unpack[_BasicCrawlerOptions],
     ) -> None:
@@ -140,6 +141,8 @@ def __init__(
             static_parser: Implementation of `AbstractHttpParser`. Parser that will be used for static crawling.
             static_crawler_specific_kwargs: `AbstractHttpCrawler` only kwargs that are passed to the sub crawler.
             playwright_crawler_specific_kwargs: `PlaywrightCrawler` only kwargs that are passed to the sub crawler.
+            save_response_cookies: Whether to save the response cookies to the session, in both the static and the
+                browser sub crawler.
             statistics: A custom `Statistics[AdaptivePlaywrightCrawlerStatisticState]` instance, allowing the use of
                 non-default configuration.
             kwargs: Additional keyword arguments to pass to the underlying `BasicCrawler`.
@@ -174,10 +177,12 @@ def __init__(
 
         static_crawler = static_crawler_class(
             parser=static_parser,
+            save_response_cookies=save_response_cookies,
             statistics=_NonPersistentStatistics(),
             **basic_crawler_kwargs_for_static_crawler,
         )
         playwright_crawler = PlaywrightCrawler(
+            save_response_cookies=save_response_cookies,
             statistics=_NonPersistentStatistics(),
             **playwright_crawler_specific_kwargs,
             **basic_crawler_kwargs_for_pw_crawler,
@@ -241,6 +246,7 @@ def with_beautifulsoup_static_parser(
         result_comparator: Callable[[RequestHandlerRunResult, RequestHandlerRunResult], bool] | None = None,
         parser_type: BeautifulSoupParserType = 'lxml',
         playwright_crawler_specific_kwargs: _PlaywrightCrawlerAdditionalOptions | None = None,
+        save_response_cookies: bool = True,
         statistics: Statistics[StatisticsState] | None = None,
         **kwargs: Unpack[_BasicCrawlerOptions],
     ) -> AdaptivePlaywrightCrawler[ParsedHttpCrawlingContext[BeautifulSoup], BeautifulSoup, Tag]:
@@ -255,6 +261,7 @@ def with_beautifulsoup_static_parser(
             result_comparator=result_comparator,
             static_parser=BeautifulSoupParser(parser=parser_type),
             playwright_crawler_specific_kwargs=playwright_crawler_specific_kwargs,
+            save_response_cookies=save_response_cookies,
             statistics=adaptive_statistics,
             **kwargs,
         )
@@ -267,6 +274,7 @@ def with_parsel_static_parser(
         result_checker: Callable[[RequestHandlerRunResult], bool] | None = None,
         result_comparator: Callable[[RequestHandlerRunResult, RequestHandlerRunResult], bool] | None = None,
         playwright_crawler_specific_kwargs: _PlaywrightCrawlerAdditionalOptions | None = None,
+        save_response_cookies: bool = True,
         statistics: Statistics[StatisticsState] | None = None,
         **kwargs: Unpack[_BasicCrawlerOptions],
     ) -> AdaptivePlaywrightCrawler[ParsedHttpCrawlingContext[Selector], Selector, Selector]:
@@ -281,6 +289,7 @@ def with_parsel_static_parser(
             result_comparator=result_comparator,
             static_parser=ParselParser(),
             playwright_crawler_specific_kwargs=playwright_crawler_specific_kwargs,
+            save_response_cookies=save_response_cookies,
             statistics=adaptive_statistics,
             **kwargs,
         )

src/crawlee/crawlers/_http/_http_crawler.py

@@ -10,7 +10,7 @@
 if TYPE_CHECKING:
     from typing import Unpack
 
-    from crawlee.crawlers import BasicCrawlerOptions
+    from crawlee.crawlers import HttpCrawlerOptions
 
 
 @docs_group('Crawlers')
@@ -48,7 +48,7 @@ async def request_handler(context: HttpCrawlingContext) -> None:
 
     def __init__(
         self,
-        **kwargs: Unpack[BasicCrawlerOptions[ParsedHttpCrawlingContext[bytes]]],
+        **kwargs: Unpack[HttpCrawlerOptions[ParsedHttpCrawlingContext[bytes]]],
     ) -> None:
         """Initialize a new instance.
 

src/crawlee/crawlers/_playwright/_playwright_crawler.py

@@ -123,6 +123,7 @@ def __init__(
         headless: bool | None = None,
         use_incognito_pages: bool | None = None,
         navigation_timeout: timedelta | None = None,
+        save_response_cookies: bool = True,
         **kwargs: Unpack[BasicCrawlerOptions[TCrawlingContext, StatisticsState]],
     ) -> None:
         """Initialize a new instance.
@@ -155,9 +156,11 @@ def __init__(
                 the request handler)
             goto_options: Additional options to pass to Playwright's `Page.goto()` method. The `timeout` option is
                 not supported, use `navigation_timeout` instead.
+            save_response_cookies: Whether to save the cookies of the page to the session after the request handler.
             kwargs: Additional keyword arguments to pass to the underlying `BasicCrawler`.
         """
         self._shared_navigation_timeouts: dict[int, SharedTimeout] = {}
+        self._save_response_cookies = save_response_cookies
 
         if browser_pool:
             # Raise an exception if browser_pool is provided together with other browser-related arguments.
@@ -584,7 +587,7 @@ async def _execute_post_navigation_hooks(self, context: TPostNavContext) -> Asyn
     async def _create_crawling_context(self, context: TPostNavContext) -> AsyncGenerator[TCrawlingContext, None]:
         yield self._build_crawling_context(context)
 
-        if context.session:
+        if context.session and self._save_response_cookies:
             pw_cookies = await self._get_cookies(context.page)
             context.session.cookies.set_cookies_from_playwright_format(pw_cookies)
 
@@ -668,3 +671,8 @@ class PlaywrightCrawlerOptions(
 
     It is intended for typing forwarded `__init__` arguments in the subclasses.
     """
+
+    # Kept out of `_PlaywrightCrawlerAdditionalOptions`, since `AdaptivePlaywrightCrawler` passes those to its browser
+    # sub-crawler only. It takes `save_response_cookies` as its own argument and passes it to both sub-crawlers.
+    save_response_cookies: NotRequired[bool]
+    """Whether to save the cookies of the page to the session after the request handler."""

src/crawlee/crawlers/_playwright/_playwright_http_client.py

@@ -71,7 +71,7 @@ async def send_request(
         try:
             # The browser context holds the cookies of the session, added before the navigation, with the changes the
             # page made since. The browser stores the response cookies there too, and the crawler saves them to the
-            # session after the request handler.
+            # session after the request handler unless `save_response_cookies` is disabled.
             if page is not None and cookies is None:
                 context_cookies = await page.context.cookies()
                 cookies = SessionCookies()

src/crawlee/crawlers/_stagehand/_stagehand_crawler.py

@@ -101,6 +101,7 @@ def __init__(
         browser_new_context_options: dict[str, Any] | None = None,
         goto_options: GotoOptions | None = None,
         navigation_timeout: timedelta | None = None,
+        save_response_cookies: bool = True,
         **kwargs: Unpack[BasicCrawlerOptions[StagehandCrawlingContext, StatisticsState]],
     ) -> None:
         """Initialize a new instance.
@@ -124,6 +125,7 @@ def __init__(
                 supported - use `navigation_timeout` instead.
             navigation_timeout: Timeout for the navigation phase (from opening the page to calling the request handler).
             Defaults to one minute.
+            save_response_cookies: Whether to save the cookies of the page to the session after the request handler.
             kwargs: Additional keyword arguments forwarded to `BasicCrawler`.
         """
         if browser_pool is not None:
@@ -164,6 +166,7 @@ def __init__(
             browser_pool=browser_pool,
             goto_options=goto_options,
             navigation_timeout=navigation_timeout,
+            save_response_cookies=save_response_cookies,
             **kwargs,
         )
 

src/crawlee/http_clients/_base.py

@@ -194,19 +194,16 @@ class HttpClient(ABC):
     def __init__(
         self,
         *,
-        persist_cookies_per_session: bool = True,
         follow_redirects: bool = True,
         headers: Mapping[str, str] | None = None,
     ) -> None:
         """Initialize a new instance.
 
         Args:
-            persist_cookies_per_session: Whether to persist cookies per HTTP session.
             follow_redirects: Whether `send_request` follows redirects. Without following them, it returns the first
                 response as it is.
             headers: The headers to include in every request. The headers passed to `send_request` win over them.
         """
-        self._persist_cookies_per_session = persist_cookies_per_session
         self._follow_redirects = follow_redirects
         self._headers = HttpHeaders(headers or {})
 
@@ -244,11 +241,9 @@ async def send_request(
             method: The HTTP method to use.
             headers: The headers to include in the request.
             payload: The data to be sent as the request body.
-            session: The session whose cookies are sent. The response cookies are stored in it when
-                `persist_cookies_per_session` is enabled. Otherwise, and without a session, they are kept only for the
-                redirect chain of this request.
-            cookies: The cookies to send and store instead of the cookies of `session`, even with
-                `persist_cookies_per_session` disabled.
+            session: The session whose cookies are sent and stored. Without a session or `cookies`, the response
+                cookies are kept only for the redirect chain of this request.
+            cookies: The cookies to send and store instead of the cookies of `session`.
             proxy_info: The information about the proxy to be used.
             timeout: Maximum time allowed to receive the response, including all redirects, and to read its body.
                 Without it, the default timeout of the HTTP client applies.
@@ -267,13 +262,7 @@ async def send_request(
         headers = self._headers | HttpHeaders(headers or {})
 
         if cookies is None:
-            if session is None:
-                cookies = SessionCookies()
-            elif self._persist_cookies_per_session:
-                cookies = session.cookies
-            else:
-                # The cookies of the session are still sent, but the response cookies go to a copy.
-                cookies = SessionCookies(session.cookies)
+            cookies = session.cookies if session is not None else SessionCookies()
 
         # `encoded=True` keeps the path and the query byte for byte, including a safe `%2F` in query.
         current_url = _with_ascii_host(URL(url, encoded=True))