D g.dZddlZddlmZddlmZmZmZmZm Z m Z ddl m Z ddl mZddlmZddlmZejd ejZejd ejZejd ejZejd ejejzZejd ejejzZejdejZdZ d-dedeedededef dZ d.dedeedefdZ! d/dededeedefdZ"ejdejZ#d.dedeedefdZ$ d0dedeedeedeedef d Z% d1dedeedeedefd!Z& d2dedeed#edeedef d$Z' d3dedeededeedef d%Z( d4ded&ededefd'Z) d5ded&eded)eede e d*e e*efff d+Z+dedefd,Z,dS)6z( Functions for dealing with markup text N)name2codepoint)IterableMatchOptionalPatternTupleUnion)urljoin) StrOrBytes)safe_url_string) to_unicodezI&((?P[a-z\d]+)|#(?P\d+)|#x(?P[a-f\d]+))(?P;?)z<[a-zA-Z\/!].*?>z5]*href\s*=\s*[\"\']\s*([^\"\'\s]+)\s*[\"\']z}]*http-equiv[^>]*refresh[^>]*content\s*=\s*(?P["\'])(?P(\d*\.)?\d+)\s*;\s*url=\s*(?P.*?)(?P=quote)z]*content\s*=\s*(?P["\'])(?P(\d*\.)?\d+)\s*;\s*url=\s*(?P.*?)(?P=quote)[^>]*?\shttp-equiv\s*=[^>]*refreshz<((?P.*?)(?P\]\]>))z Tutf-8textkeepremove_illegalencodingreturncdttdtffd }t|t ||S)uRemove entities from the given `text` by converting them to their corresponding unicode character. `text` can be a unicode string or a byte string encoded in the given `encoding` (which defaults to 'utf-8'). If `keep` is passed (with a list of entity names) those entities will be kept (they won't be removed). It supports both numeric entities (``&#nnnn;`` and ``&#hhhh;``) and named entities (such as `` `` or ``>``). If `remove_illegal` is ``True``, entities that can't be converted are removed. If `remove_illegal` is ``False``, entities that can't be converted are kept "as is". For more information see the tests. Always returns a unicode string (with the entities removed). >>> import w3lib.html >>> w3lib.html.replace_entities(b'Price: £100') 'Price: \xa3100' >>> print(w3lib.html.replace_entities(b'Price: £100')) Price: £100 >>> mrc|}d}|drt|dd}n|drt|dd}n|drm|d}|vr|dSt j|p%t j|}|Z d|cxkrdkr&nn#t |fd St|S#ttf$rYnwxYwr|d rd n|dS) Ndec hexnamedrcp1252 semicolon) groupdictgetintlowergrouprbytesdecodechr ValueError OverflowError)rgroupsnumber entity_namerrs ?/var/www/sysmax/venv/lib/python3.11/site-packages/w3lib/html.pyconvert_entityz(replace_entities..convert_entityEs ::e   ++FF ZZ   ++FF ZZ   /K  ""d**wwqzz!'+K88N>> import w3lib.html >>> w3lib.html.replace_tags('This text contains some tag') 'This text contains some tag' >>> w3lib.html.replace_tags('

Je ne parle pas fran\xe7ais

', ' -- ', 'latin-1') ' -- Je ne parle pas -- fran\xe7ais -- -- ' >>> )_tag_rer3r )rr;rs r/ replace_tagsr>js", ;;ujx88 9 99r5z|$)cXt||}td|S)zRemove HTML Comments. >>> import w3lib.html >>> w3lib.html.remove_comments(b"test whatever") 'test whatever' >>> r!)r _REMOVECOMMENTS_REr3)rrutexts r/remove_commentsrBs) tX & &E  ! !"e , ,,r5 which_onescjrrtddDdDdtdtffd dttdtffd }d }t j|tjtjz}||t||S) a;Remove HTML Tags only. `which_ones` and `keep` are both tuples, there are four cases: ============== ============= ========================================== ``which_ones`` ``keep`` what it does ============== ============= ========================================== **not empty** empty remove all tags in ``which_ones`` empty **not empty** remove all tags except the ones in ``keep`` empty empty remove all tags **not empty** **not empty** not allowed ============== ============= ========================================== Remove all tags: >>> import w3lib.html >>> doc = '

This is a link: example

' >>> w3lib.html.remove_tags(doc) 'This is a link: example' >>> Keep only some tags: >>> w3lib.html.remove_tags(doc, keep=('div',)) '
This is a link: example
' >>> Remove only specific tags: >>> w3lib.html.remove_tags(doc, which_ones=('a','b')) '

This is a link: example

' >>> You can't remove some and keep some: >>> w3lib.html.remove_tags(doc, which_ones=('a',), keep=('p',)) Traceback (most recent call last): ... ValueError: Cannot use both which_ones and keep >>> z#Cannot use both which_ones and keepc6h|]}|Srr%.0tags r/ zremove_tags..s 444##))++444r5c6h|]}|SrrFrGs r/rJzremove_tags..s ( ( (CCIIKK ( ( (r5rIrc@|}r|vS|vSr7rF)rIrrCs r/ will_removez remove_tags..will_removes-iikk  #*$ $d? "r5rcr|d}|rdn|dS)Nr!r)r&)rrIrMs r/ remove_tagzremove_tags..remove_tags5ggajj [%%5rr1771::5r5z/]+).*?>) r*r1r8rrecompileDOTALL IGNORECASEr3r )rrCrrrPregexretagsrMs `` @r/ remove_tagsrWsb@d@>???44444J ( (4 ( ( (D#########6eCj6S666666 E Zry2=8 9 9F ::j*T8"<"< = ==r5ct||}|radd|D}tj|tjtjz}|d|}|S)aRemove tags and their content. `which_ones` is a tuple of which tags to remove including their content. If is empty, returns the string unmodified. >>> import w3lib.html >>> doc = '

This is a link: example

' >>> w3lib.html.remove_tags_with_content(doc, which_ones=('b',)) '' >>> |c &g|]}d|d|d|dS)|rrGs r/ z,remove_tags_with_content..s4SSS._get_fragmentss}%%c**  E$zz!}} GWfWn% % % %KKKFF&''lr5r!)rrcdata_d) r1rrr rr _cdata_re isinstancer4r&)rrrrrsrAret_textfragments r/unquote_markupry s"   "3<  %U3Z( )     tX & &EH"N5)4422 h $ $ 2 (tN HH y11 1HH Or5baseurlct||}t|}|r?tt |t |d|St |S)zReturn the base url if declared in the given HTML `text`, relative to the given base url. If no base url is found, the given `baseurl` is returned. )rrO)rB _baseurl_rer9r r r&)rrzrrArs r/ get_base_urlr}4sw!999E5!!A( G $ $oaggajj8&T&T&T   w'''r5scriptnoscript ignore_tagsNNc t||}n#t$rt|wxYwt||}t t |}t |pt|}|rlt| d}t| d d|}t||}||fSdS)aXReturn the http-equiv parameter of the HTML meta element from the given HTML text and return a tuple ``(interval, url)`` where interval is an integer containing the delay in seconds (or zero if not present) and url is a string with the absolute url to redirect. If no meta redirect is found, ``(None, None)`` is returned. r$urlz "'r)r UnicodeDecodeErrorprintr_rBr4_meta_refresh_rer9_meta_refresh_re2floatr&r stripr )rrzrrrArintervalrs r/get_meta_refreshrHs4**  d   %UK 8 8E ,U33 4 4E&&I*;*B*B5*I*IA((aggenn226::HEEgs##}zs.c6|tS)a Strip all leading and trailing space characters (as defined in https://www.w3.org/TR/html5/infrastructure.html#space-character). Such stripping is useful e.g. for processing HTML element attributes which contain URLs, like ``href``, ``src`` or form ``action`` - HTML5 standard defines them as "valid URL potentially surrounded by spaces" or "valid non-empty URL potentially surrounded by spaces". >>> strip_html5_whitespace(' hello\n') 'hello' )rHTML5_WHITESPACE)rs r/strip_html5_whitespacerhs ::& ' ''r5)rTrr7)r!N)rrN)rN)r`r!N)rTN)r!r)r!rr~)-__doc__rQ html.entitiesrtypingrrrrrr urllib.parser w3lib._typesr w3lib.urlr w3lib.utilr rRrTr2rSr=Ir|rrrurr1r8r4r:r>r@rBrWr_rhryr}rrrrr5r/rs ((((((CCCCCCCCCCCCCCCC ######%%%%%%!!!!!! "*PM   "*(") 4 4bjQSUSWXX 2:EI BJLI  BJCRY   !  ?C?C ?C 3-?C?C ?C  ?C?C?C?CD<> - -* - - - - - - !#" E>E> E> E> 3-E>sm E>  E>E>E>E>RQU "*3-@H 4!3"   sm   0" '' ' 3-''sm '  ''''VAH(( ()(:=(((((,!7   #   5 eE3J/ /0 @ ( ( ( ( ( ( ( (r5