[pull] main from python:main

Doc/c-api/file.rst

@@ -131,3 +131,22 @@ the :mod:`io` APIs instead.
 
    Write string *s* to file object *p*.  Return ``0`` on success or ``-1`` on
    failure; the appropriate exception will be set.
+
+
+Deprecated API
+^^^^^^^^^^^^^^
+
+
+These are :term:`soft deprecated` APIs that were included in Python's C API
+by mistake. They are documented solely for completeness; use other
+``PyFile*`` APIs instead.
+
+.. c:function:: PyObject *PyFile_NewStdPrinter(int fd)
+
+   Use :c:func:`PyFile_FromFd` with defaults (``fd, NULL, "w", -1, NULL, NULL, NULL, 0``) instead.
+
+.. c:var:: PyTypeObject PyStdPrinter_Type
+
+   Type of file-like objects used internally at Python startup when :py:mod:`io` is
+   not yet available.
+   Use Python :py:func:`open` or :c:func:`PyFile_FromFd` to create file objects instead.

Doc/c-api/gen.rst

@@ -45,6 +45,7 @@ than explicitly calling :c:func:`PyGen_New` or :c:func:`PyGen_NewWithQualName`.
    A reference to *frame* is stolen by this function.  The *frame* argument
    must not be ``NULL``.
 
+
 .. c:function:: PyCodeObject* PyGen_GetCode(PyGenObject *gen)
 
    Return a new :term:`strong reference` to the code object wrapped by *gen*.
@@ -82,3 +83,14 @@ Asynchronous Generator Objects
    This function always succeeds.
 
    .. versionadded:: 3.6
+
+
+Deprecated API
+^^^^^^^^^^^^^^
+
+.. c:macro:: PyAsyncGenASend_CheckExact(op)
+
+   This is a :term:`soft deprecated` API that was included in Python's C API
+   by mistake.
+
+   It is solely here for completeness; do not use this API.

Doc/c-api/long.rst

@@ -855,3 +855,31 @@ The :c:type:`PyLongWriter` API can be used to import an integer.
    If *writer* is ``NULL``, no operation is performed.
 
    The writer instance and the *digits* array are invalid after the call.
+
+
+Deprecated API
+^^^^^^^^^^^^^^
+
+These macros are :term:`soft deprecated`. They describe parameters
+of the internal representation of :c:type:`PyLongObject` instances.
+
+Use :c:func:`PyLong_GetNativeLayout` instead, along with :c:func:`PyLong_Export`
+to read integer data or :c:type:`PyLongWriter` to write it.
+These currently use the same layout, but are designed to continue working correctly
+even if CPython's internal integer representation changes.
+
+
+.. c:macro:: PyLong_SHIFT
+
+   This is equivalent to :c:member:`~PyLongLayout.bits_per_digit` in
+   the output of :c:func:`PyLong_GetNativeLayout`.
+
+
+.. c:macro:: PyLong_BASE
+
+   This is currently equivalent to :c:expr:`1 << PyLong_SHIFT`.
+
+
+.. c:macro:: PyLong_MASK
+
+   This is currently equivalent to :c:expr:`(1 << PyLong_SHIFT) - 1`

Doc/c-api/set.rst

@@ -166,3 +166,20 @@ subtypes but not for instances of :class:`frozenset` or its subtypes.
    Empty an existing set of all elements. Return ``0`` on
    success. Return ``-1`` and raise :exc:`SystemError` if *set* is not an instance of
    :class:`set` or its subtype.
+
+
+Deprecated API
+^^^^^^^^^^^^^^
+
+.. c:macro:: PySet_MINSIZE
+
+   A :term:`soft deprecated` constant representing the size of an internal
+   preallocated table inside :c:type:`PySetObject` instances.
+
+   This is documented solely for completeness, as there are no guarantees
+   that a given version of CPython uses preallocated tables with a fixed
+   size.
+   In code that does not deal with unstable set internals,
+   :c:macro:`!PySet_MINSIZE` can be replaced with a small constant like ``8``.
+
+   If looking for the size of a set, use :c:func:`PySet_Size` instead.

Doc/library/base64.rst

@@ -51,7 +51,7 @@ The :rfc:`4648` encodings are suitable for encoding binary data so that it can b
 safely sent by email, used as parts of URLs, or included as part of an HTTP
 POST request.
 
-.. function:: b64encode(s, altchars=None)
+.. function:: b64encode(s, altchars=None, *, wrapcol=0)
 
    Encode the :term:`bytes-like object` *s* using Base64 and return the encoded
    :class:`bytes`.
@@ -61,9 +61,16 @@ POST request.
    This allows an application to e.g. generate URL or filesystem safe Base64
    strings.  The default is ``None``, for which the standard Base64 alphabet is used.
 
+   If *wrapcol* is non-zero, insert a newline (``b'\n'``) character
+   after at most every *wrapcol* characters.
+   If *wrapcol* is zero (default), do not insert any newlines.
+
    May assert or raise a :exc:`ValueError` if the length of *altchars* is not 2.  Raises a
    :exc:`TypeError` if *altchars* is not a :term:`bytes-like object`.
 
+   .. versionchanged:: next
+      Added the *wrapcol* parameter.
+
 
 .. function:: b64decode(s, altchars=None, validate=False)
 
@@ -214,9 +221,9 @@ Refer to the documentation of the individual functions for more information.
    instead of 4 consecutive spaces (ASCII 0x20) as supported by 'btoa'. This
    feature is not supported by the "standard" Ascii85 encoding.
 
-   *wrapcol* controls whether the output should have newline (``b'\n'``)
-   characters added to it. If this is non-zero, each output line will be
-   at most this many characters long, excluding the trailing newline.
+   If *wrapcol* is non-zero, insert a newline (``b'\n'``) character
+   after at most every *wrapcol* characters.
+   If *wrapcol* is zero (default), do not insert any newlines.
 
    *pad* controls whether the input is padded to a multiple of 4
    before encoding. Note that the ``btoa`` implementation always pads.

Doc/library/binascii.rst

@@ -58,7 +58,7 @@ The :mod:`binascii` module defines the following functions:
 
    Valid base64:
 
-   * Conforms to :rfc:`3548`.
+   * Conforms to :rfc:`4648`.
    * Contains only characters from the base64 alphabet.
    * Contains no excess data after padding (including excess padding, newlines, etc.).
    * Does not start with a padding.
@@ -67,15 +67,24 @@ The :mod:`binascii` module defines the following functions:
       Added the *strict_mode* parameter.
 
 
-.. function:: b2a_base64(data, *, newline=True)
+.. function:: b2a_base64(data, *, wrapcol=0, newline=True)
 
-   Convert binary data to a line of ASCII characters in base64 coding. The return
-   value is the converted line, including a newline char if *newline* is
-   true.  The output of this function conforms to :rfc:`3548`.
+   Convert binary data to a line(s) of ASCII characters in base64 coding,
+   as specified in :rfc:`4648`.
+
+   If *wrapcol* is non-zero, insert a newline (``b'\n'``) character
+   after at most every *wrapcol* characters.
+   If *wrapcol* is zero (default), do not insert any newlines.
+
+   If *newline* is true (default), a newline character will be added
+   at the end of the output.
 
    .. versionchanged:: 3.6
       Added the *newline* parameter.
 
+   .. versionchanged:: next
+      Added the *wrapcol* parameter.
+
 
 .. function:: a2b_qp(data, header=False)
 

Doc/library/unicodedata.rst

@@ -184,6 +184,28 @@ following functions:
       '0041 0303'
 
 
+.. function:: grapheme_cluster_break(chr, /)
+
+   Returns the Grapheme_Cluster_Break property assigned to the character.
+
+   .. versionadded:: next
+
+
+.. function:: indic_conjunct_break(chr, /)
+
+   Returns the Indic_Conjunct_Break property assigned to the character.
+
+   .. versionadded:: next
+
+
+.. function:: extended_pictographic(chr, /)
+
+   Returns ``True`` if the character has the Extended_Pictographic property,
+   ``False`` otherwise.
+
+   .. versionadded:: next
+
+
 .. function:: normalize(form, unistr, /)
 
    Return the normal form *form* for the Unicode string *unistr*. Valid values for
@@ -225,6 +247,24 @@ following functions:
    .. versionadded:: 3.8
 
 
+.. function:: iter_graphemes(unistr, start=0, end=sys.maxsize, /)
+
+   Returns an iterator to iterate over grapheme clusters.
+   With optional *start*, iteration begins at that position.
+   With optional *end*, iteration stops at that position.
+
+   Converting an emitted item to string returns a substring corresponding to
+   the grapheme cluster.
+   Its ``start`` and ``end`` attributes denote the start and end of
+   the grapheme cluster.
+
+   It uses extended grapheme cluster rules defined by Unicode
+   Standard Annex #29, `"Unicode Text Segmentation"
+   <https://www.unicode.org/reports/tr29/>`_.
+
+   .. versionadded:: next
+
+
 In addition, the module exposes the following constant:
 
 .. data:: unidata_version
@@ -234,7 +274,7 @@ In addition, the module exposes the following constant:
 
 .. data:: ucd_3_2_0
 
-   This is an object that has the same methods as the entire module, but uses the
+   This is an object that has most of the methods of the entire module, but uses the
    Unicode database version 3.2 instead, for applications that require this
    specific version of the Unicode database (such as IDNA).
 

Doc/whatsnew/3.15.rst

@@ -435,12 +435,22 @@ argparse
   inline code when color output is enabled.
   (Contributed by Savannah Ostrowski in :gh:`142390`.)
 
-base64 & binascii
------------------
+base64
+------
+
+* Added the *pad* parameter in :func:`~base64.z85encode`.
+  (Contributed by Hauke Dämpfling in :gh:`143103`.)
+
+* Added the *wrapcol* parameter in :func:`~base64.b64encode`.
+  (Contributed by Serhiy Storchaka in :gh:`143214`.)
+
+
+binascii
+--------
+
+* Added the *wrapcol* parameter in :func:`~binascii.b2a_base64`.
+  (Contributed by Serhiy Storchaka in :gh:`143214`.)
 
-* CPython's underlying base64 implementation now encodes 2x faster and decodes 3x
-  faster thanks to simple CPU pipelining optimizations.
-  (Contributed by Gregory P. Smith & Serhiy Storchaka in :gh:`143262`.)
 
 calendar
 --------
@@ -801,6 +811,16 @@ unicodedata
   `Unicode Standard Annex #31 <https://www.unicode.org/reports/tr31/>`_ identifier.
   (Contributed by Stan Ulbrych in :gh:`129117`.)
 
+* Add the :func:`~unicodedata.iter_graphemes`
+  function to iterate over grapheme clusters according to rules defined in
+  `Unicode Standard Annex #29, "Unicode Text Segmentation"
+  <https://www.unicode.org/reports/tr29/>`_.
+  Add :func:`~unicodedata.grapheme_cluster_break`,
+  :func:`~unicodedata.indic_conjunct_break` and
+  :func:`~unicodedata.extended_pictographic` functions to get the properties
+  of the character which are related to the above algorithm.
+  (Contributed by Serhiy Storchaka and Guillaume Sanchez in :gh:`74902`.)
+
 
 unittest
 --------
@@ -878,6 +898,13 @@ Optimizations
   (Contributed by Chris Eibl, Ken Jin, and Brandt Bucher in :gh:`143068`.
   Special thanks to the MSVC team including Hulon Jenkins.)
 
+base64 & binascii
+-----------------
+
+* CPython's underlying base64 implementation now encodes 2x faster and decodes 3x
+  faster thanks to simple CPU pipelining optimizations.
+  (Contributed by Gregory P. Smith and Serhiy Storchaka in :gh:`143262`.)
+
 csv
 ---
 

Include/internal/pycore_global_strings.h

@@ -865,6 +865,7 @@ struct _Py_global_strings {
         STRUCT_FOR_ID(which)
         STRUCT_FOR_ID(who)
         STRUCT_FOR_ID(withdata)
+        STRUCT_FOR_ID(wrapcol)
         STRUCT_FOR_ID(writable)
         STRUCT_FOR_ID(write)
         STRUCT_FOR_ID(write_through)

Include/internal/pycore_optimizer.h

@@ -233,7 +233,7 @@ _PyJit_TryInitializeTracing(PyThreadState *tstate, _PyInterpreterFrame *frame,
     _Py_CODEUNIT *close_loop_instr, int curr_stackdepth, int chain_depth, _PyExitData *exit,
     int oparg, _PyExecutorObject *current_executor);
 
-void _PyJit_FinalizeTracing(PyThreadState *tstate);
+void _PyJit_FinalizeTracing(PyThreadState *tstate, int err);
 void _PyJit_TracerFree(_PyThreadStateImpl *_tstate);
 
 void _PyJit_Tracer_InvalidateDependency(PyThreadState *old_tstate, void *obj);

Include/internal/pycore_tstate.h

@@ -54,6 +54,7 @@ typedef struct _PyJitTracerTranslatorState {
 } _PyJitTracerTranslatorState;
 
 typedef struct _PyJitTracerState {
+    bool is_tracing;
     _PyJitTracerInitialState initial_state;
     _PyJitTracerPreviousState prev_state;
     _PyJitTracerTranslatorState translator_state;

Lib/base64.py

@@ -45,14 +45,17 @@ def _bytes_from_decode_data(s):
 
 # Base64 encoding/decoding uses binascii
 
-def b64encode(s, altchars=None):
+def b64encode(s, altchars=None, *, wrapcol=0):
     """Encode the bytes-like object s using Base64 and return a bytes object.
 
     Optional altchars should be a byte string of length 2 which specifies an
     alternative alphabet for the '+' and '/' characters.  This allows an
     application to e.g. generate url or filesystem safe Base64 strings.
+
+    If wrapcol is non-zero, insert a newline (b'\\n') character after at most
+    every wrapcol characters.
     """
-    encoded = binascii.b2a_base64(s, newline=False)
+    encoded = binascii.b2a_base64(s, wrapcol=wrapcol, newline=False)
     if altchars is not None:
         assert len(altchars) == 2, repr(altchars)
         return encoded.translate(bytes.maketrans(b'+/', altchars))
@@ -327,9 +330,8 @@ def a85encode(b, *, foldspaces=False, wrapcol=0, pad=False, adobe=False):
     instead of 4 consecutive spaces (ASCII 0x20) as supported by 'btoa'. This
     feature is not supported by the "standard" Adobe encoding.
 
-    wrapcol controls whether the output should have newline (b'\\n') characters
-    added to it. If this is non-zero, each output line will be at most this
-    many characters long, excluding the trailing newline.
+    If wrapcol is non-zero, insert a newline (b'\\n') character after at most
+    every wrapcol characters.
 
     pad controls whether the input is padded to a multiple of 4 before
     encoding. Note that the btoa implementation always pads.
@@ -566,11 +568,10 @@ def encodebytes(s):
     """Encode a bytestring into a bytes object containing multiple lines
     of base-64 data."""
     _input_type_check(s)
-    pieces = []
-    for i in range(0, len(s), MAXBINSIZE):
-        chunk = s[i : i + MAXBINSIZE]
-        pieces.append(binascii.b2a_base64(chunk))
-    return b"".join(pieces)
+    result = binascii.b2a_base64(s, wrapcol=MAXLINESIZE)
+    if result == b'\n':
+        return b''
+    return result
 
 
 def decodebytes(s):