U i @sUddlmZddlZddlmZddlmZddlmZm Z m Z m Z ddl m Z mZmZmZmZddlmZdd lmZmZdd lmZmZmZmZmZmZed Ze Z!e!"e#d gZ$d e%d<gZ&d e%d<e D]HZ'z"ee're$(e'n e&(e'Wqe)k r"e&(e'YqXqe$e&Z*d e%d<d,dddddddddddd ddZ+d-d ddddddddddd! d"d#Z,d.d$ddddddddddd% d&d'Z-d/d(ddddddddddd) d*d+Z.dS)0) annotationsN)PathLike)BinaryIO)coherence_ratioencoding_languagesmb_encoding_languagesmerge_coherence_ratios)IANA_SUPPORTEDIANA_SUPPORTED_SIMILARTOO_BIG_SEQUENCETOO_SMALL_SEQUENCETRACE) mess_ratio) CharsetMatchCharsetMatches)any_specified_encodingcut_sequence_chunks iana_nameidentify_sig_or_bomis_multi_byte_encodingshould_strip_sig_or_bomZcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)sz list[str] _mb_supported _sb_supportedIANA_SUPPORTED_MB_FIRST皙?TF皙?zbytes | bytearrayintfloatzlist[str] | Noneboolr) sequencessteps chunk_size threshold cp_isolation cp_exclusionpreemptive_behaviourexplainlanguage_thresholdenable_fallbackreturnc @ Cs t|ttfs tdt||r>tj} tt t t t |} | dkrt d|rptt t | tt|dddgdgS|dk rtt d d |d d |D}ng}|dk rtt d d |dd |D}ng}| ||krtt d||| d}| }|dkr4| ||kr4t| |}t |tk} t |tk} | rftt d| n| r~tt d| g}|rt|nd}|dk r||tt d|t}g}g}t}t}i}d}t}d}d}d}d}d}d}t}t}t|\} }!| dk r6|| tt dt |!| |dd|krT|d|tD ]}"|rv|"|krvq\|r|"|krq\|"|krq\||"d}#| |"k}$|$ot|"}%|"dkr|$stt d|"q\|"dkr|$stt d|"q\|"|krtt d|"q\|"|kr8tt d|"q\z t|"}&Wn.ttfk rrtt d|"Yq\YnX|r|&stt |"}'n tt!|"}'|'"|stt d|"|'|q\|r|&s||krtt d|"||q\|r|&stt d |"q\z| rP|&dkrPt#|%dkr4|dtd!n|t |!td!|"d"nd|"d#kr|$rt#||"d"}#|#r|#dd$kr|#dd}#n&t#|%dkr|n|t |!d|"d"}#Wn\t$t%fk r}(z8t|(t%stt d%|"t#|(||"WYq\W5d}(~(XYnXt&|$s dnt |!| t| |})|&oP|#dk oPt |#| k}*|*rftt d&|"|#dk rP|&sPt'|#}+|(|+},|,dk rP|,\}-}.}/|/rt||"|-|$|.| dks|"|ddfkr|#nd|d'}0||0||"tt d(|"t)|-d)d*d+|"|ddfkrj|-d,krj|-dkr`t d-|0j*|rRtt t | t|0gS||0t |r\|dks||kr\d|kr\d|kr\|+}1t d-|1j*|rtt t | t|1gSq\np||"tt d.|"| r\|"dd|d/d0fkr\t||"||$g|#|d'}2|"|kr8|2}n|"dkrH|2}n|2}q\tt |)d1}3t,|3d2}3d}4d}5g}6g}7zt-||"|)||$|%|!|&|# D]|}8|6|8|7t.|8||d3kodt |kod2kn|7d4|kr|4d7}4|4|3ks|$r|%dkrqqWn@t$k rT}(z tt d5|"t#|(|3}4d3}5W5d}(~(XYnX|5s| r|&sz|td6dj/|"d7d8WnLt$k r}(z,tt d9|"t#|(||"WYq\W5d}(~(XYnX|7rt0|7t |7nd}9|9|k s|4|3k r||"|"t1k r$|2t1|"|#dk rJ|& sJ|3t'|#|9gdftt d:|"|4t)|9d)d*d+| r\|"dd|d/d0fkr\|5s\t||"||$g|#|d'}2|"|k r|2}n|"dk r|2}n|2}q\tt d;|"t)|9d)d*d+|& st |"}:nt!|"}:|: rtt d<|"t#|:g};|"dk rb|6D],}8t4|8||: rDd=|:nd}<|;|< q*t5|;}=nt5|;}=|= rtt d>|=|"t||"|9|$|=| dk s|"|ddfk r|#nd|d'}>||>|#dk r|& s|3t'|#|9|=d3f| r|& s|9d?k r|d7}|"|ddfk rj|9d,k rj|9dk r`t d-|>j*| rRtt t | t|>gS||>t | r|dk s||k rd|k rd|k r|+}1t d-|1j*| rtt t | t|1gS| sD|& sD|= rt,d@dA|=DddBnd}?|?dCk rDd|k rDd|k rDd3}|2|:tt dD|"|9|?| s|& r|* r|#dk rt |#| dEk r|"dFk rd|k rd|k rd3}tt dG|"|9t |#| t |#| d)|"| kr\t dH|"| rtt t | t||"gSq\t |dk r| s | s | r,tt dI| rLt dJ|j*||nd| r\|dk s| rv| rv|j6|j6k s|dk rt dK||n| rt dL||| rt dM|+j*t |dn t dN| rtt t | |S)Oaf Given a raw bytes sequence, return the best possibles charset usable to render str objects. If there is no results, it is a strong indicator that the source is binary/not text. By default, the process will extract 5 blocks of 512o each to assess the mess and coherence of a given sequence. And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will. The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page but never take it for granted. Can improve the performance. You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that purpose. This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32. By default the library does not setup any handler other than the NullHandler, if you choose to set the 'explain' toggle to True it will alter the logger configuration to add a StreamHandler that is suitable for debugging. Custom logging format and handler can be set manually. z3Expected object of type bytes or bytearray, got: {}rzsszfrom_bytes..zacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.cSsg|]}t|dqSr/r0r1r4r4r5r6~sz^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.rz>Trying to detect encoding from a tiny portion of ({}) byte(s).zIUsing lazy str decoding because the payload is quite large, ({}) byte(s).z@Detected declarative mark in sequence. Priority +1 given for %s.zIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.ascii>utf_32utf_16z\Encoding %s won't be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.>utf_7zREncoding %s won't be tested as-is because detection is unreliable without BOM/SIG.zY%s is deemed too similar to a code page that was already considered unsuited. Continuing!zESkipping %s: already fast-tracked from a similar successful encoding.z2Encoding %s does not provide an IncrementalDecoderzbSkipping %s: definitive match already found, this encoding targets different languages (%s vs %s).zXSkipping %s: already accumulated %d same-family results after definitive match (cap=%d).zCSkipping single-byte %s: multi-byte definitive match already found.gA)encodingr;uz9Code page %s does not fit given bytes sequence at ALL. %szpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes.)Zpreemptive_declarationzM%s fast-tracked (identical decoded payload to a prior encoding, chaos=%f %%).d)ndigitsrz.Encoding detection: %s is most likely the one.zZ%s fast-skipped (identical decoded payload to a prior encoding that failed chaos probing).r:r9TzaLazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %sgj@strict)errorsz^LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %szc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.z=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {},z We detected language {} using {}g{Gz?css|]\}}|VqdS)Nr4)r2_vr4r4r5 szfrom_bytes..)defaultg?zyDefinitive match found: %s (chaos=%.3f, coherence=%.2f). Encodings targeting different language families will be skipped.g\(\?> r9 utf_32_ler;r: utf_8_sig utf_32_be utf_16_be utf_16_ler-zjMulti-byte definitive match: %s (chaos=%.3f, decoded=%d/%d=%.1f%%). Single-byte encodings will be skipped.zoEncoding detection: %s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.zONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z7Encoding detection: %s will be used as a fallback matchz:Encoding detection: utf_8 will be used as a fallback matchz:Encoding detection: ascii will be used as a fallback matchz]Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.z=Encoding detection: Unable to determine any suitable charset.)7 isinstance bytearraybytes TypeErrorformattypeloggerlevel addHandlerexplain_handlersetLevelrlendebug removeHandlerrrlogjoinrr r rappendsetrraddrrModuleNotFoundError ImportErrorrr intersectionstrUnicodeDecodeError LookupErrorrangehashgetroundr<bestmaxrrdecodesumr update setdefaultrr fingerprint)@r"r#r$r%r&r'r(r)r*r+Zprevious_logger_levellengthZis_too_small_sequenceZis_too_large_sequenceZprioritized_encodingsZspecified_encodingZtestedZtested_but_hard_failureZtested_but_soft_failureZsoft_failure_skipZsuccess_fast_trackedZpayload_result_cacheZdefinitive_match_foundZdefinitive_target_languagesZ post_definitive_sb_success_countZPOST_DEFINITIVE_SB_CAPZmb_definitive_match_foundZfallback_asciiZ fallback_u8Zfallback_specifiedresultsZearly_stop_resultsZ sig_encodingZ sig_payloadZ encoding_ianaZdecoded_payloadZbom_or_sig_availableZstrip_sig_or_bomZis_multi_byte_decoderZ enc_languageseZr_Zmulti_byte_bonusZ payload_hashcachedZ cached_messZ cached_cdZ cached_passedZ fast_matchZprobable_resultZfallback_entryZmax_chunk_gave_upZearly_stop_countZlazy_str_hard_failureZ md_chunksZ md_ratioschunkZmean_mess_ratioZtarget_languagesZ cd_ratiosZchunk_languagesZcd_ratios_mergedZ current_matchZbest_coherencer4r4r5 from_bytes9s                                                  &                                     rxr) fpr#r$r%r&r'r(r)r*r+r,c Cst|||||||||| S)z Same thing than the function from_bytes but using a file pointer that is already ready. Will not close the file pointer. )rxread) ryr#r$r%r&r'r(r)r*r+r4r4r5from_fp`sr{zstr | bytes | PathLike) pathr#r$r%r&r'r(r)r*r+r,c Cs<t|d(} t| ||||||||| W5QRSQRXdS)z Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode. Can raise IOError. rbN)openr{) r|r#r$r%r&r'r(r)r*r+ryr4r4r5 from_path~s rz!PathLike | str | BinaryIO | bytes) fp_or_path_or_payloadr#r$r%r&r'r(r)r*r+r,c Cszt|ttfr,t|||||||||| d } nHt|ttfrXt|||||||||| d } nt|||||||||| d } | S)a) Detect if the given input (file, bytes, or path) points to a binary file. aka. not a string. Based on the same main heuristic algorithms and default kwargs at the sole exception that fallbacks match are disabled to be stricter around ASCII-compatible but unlikely to be a string. ) r#r$r%r&r'r(r)r*r+)rOrerrrQrPrxr{) rr#r$r%r&r'r(r)r*r+Zguessesr4r4r5 is_binarysX   r) rrrNNTFrT) rrrNNTFrT) rrrNNTFrT) rrrNNTFrF)/ __future__rloggingosrtypingrZcdrrrr Zconstantr r r r rZmdrmodelsrrutilsrrrrrr getLoggerrU StreamHandlerrX setFormatter Formatterr__annotations__rZ_supported_encr_rcrrxr{rrr4r4r4r5s        $/$ $!