.. _aerospike.Scan:

.. currentmodule:: aerospike

======================================
:class:`aerospike.Scan` --- Scan Class
======================================

.. deprecated:: 7.0.0 :class:`aerospike.Query` should be used instead.

Overview
========

The Scan object is used to return all the records in a specified set (which \
can be omitted or :py:obj:`None`). A Scan with a :py:obj:`None` set returns all the \
records in the namespace.

The scan is invoked using :meth:`foreach`, :meth:`results`, or :meth:`execute_background`. The \
bins returned can be filtered using :meth:`select`.

.. seealso::
    `Manage queries <https://aerospike.com/docs/database/manage/cluster/queries/>`_.

Fields
======

.. class:: Scan

    ttl (:class:`int`)
        The time-to-live (expiration) of the record in seconds. Note that ttl
        is only used on background scan writes.

        If this is set to :data:`aerospike.TTL_CLIENT_DEFAULT`, the scan will use the
        client's default scan policy ttl.

        See :ref:`TTL_CONSTANTS` for special values that can be set in the record ttl.

        Default: ``0`` (no limit)

        .. note::
            Requires server version >= 6.0.0

Methods
=======

.. class:: Scan
    :noindex:

    .. deprecated:: 7.0.0 :class:`aerospike.Query` should be used instead.

    .. method:: select(bin1[, bin2[, bin3..]])

        Set a filter on the record bins resulting from :meth:`results` or \
        :meth:`foreach`. If a selected bin does not exist in a record it will \
        not appear in the *bins* portion of that record tuple.


    .. method:: apply(module, function[, arguments])

        Apply a record UDF to each record found by the scan \
        `User-defined functions (UDFs) <https://aerospike.com/docs/database/learn/architecture/udf/>`_.

        :param str module: the name of the Lua module.
        :param str function: the name of the Lua function within the *module*.
        :param list arguments: optional arguments to pass to the *function*. NOTE: these arguments must be types supported by Aerospike See: `supported data types <https://aerospike.com/docs/develop/data-types/scalar>`_.
            If you need to use an unsupported type, (e.g. set or tuple) you must use your own serializer.
        :return: one of the supported types, :class:`int`, :class:`str`, :class:`float` (double), :class:`list`, :class:`dict` (map), :class:`bytearray` (bytes), :class:`bool`.

        .. seealso:: `Developing Record UDFs <https://aerospike.com/docs/database/advanced/udf/modules/record/develop>`_


    .. method:: add_ops(ops)

        Add a list of write ops to the scan.
        When used with :meth:`Scan.execute_background` the scan will perform the write ops on any records found.
        If no predicate is attached to the scan it will apply ops to all the records in the specified set. See :mod:`aerospike_helpers` for available ops.

        :param ops: `list` A list of write operations generated by the aerospike_helpers e.g. list_operations, map_operations, etc.

        .. note::
            Requires server version >= 4.7.0.

        .. code-block:: python

            import aerospike
            from aerospike_helpers.operations import list_operations
            from aerospike_helpers.operations import operations
            scan = client.scan('test', 'demo')

            ops =  [
                operations.append(test_bin, 'val_to_append'),
                list_operations.list_remove_by_index(test_bin, list_index_to_remove, aerospike.LIST_RETURN_NONE)
            ]
            scan.add_ops(ops)

            id = scan.execute_background()
            client.close()

        For a more comprehensive example, see using a list of write ops with :meth:`Query.execute_background` .


    .. method:: results([policy[, nodename]]) -> list of (key, meta, bins)

        Buffer the records resulting from the scan, and return them as a \
        :class:`list` of records.

        :param dict policy: optional :ref:`aerospike_scan_policies`.
        :param str nodename: optional Node ID of node used to limit the scan to a single node.

        :return: a :class:`list` of :ref:`aerospike_record_tuple`.


        .. code-block:: python

            import aerospike
            import pprint

            pp = pprint.PrettyPrinter(indent=2)
            config = { 'hosts': [ ('127.0.0.1',3000)]}
            client = aerospike.client(config)

            client.put(('test','test','key1'), {'id':1,'a':1},
                policy={'key':aerospike.POLICY_KEY_SEND})
            client.put(('test','test','key2'), {'id':2,'b':2},
                policy={'key':aerospike.POLICY_KEY_SEND})

            scan = client.scan('test', 'test')
            scan.select('id','a','zzz')
            res = scan.results()
            pp.pprint(res)
            client.close()

        .. note::

            We expect to see:

            .. code-block:: python

                [ ( ( 'test',
                      'test',
                      u'key2',
                      bytearray(b'\xb2\x18\n\xd4\xce\xd8\xba:\x96s\xf5\x9ba\xf1j\xa7t\xeem\x01')),
                    { 'gen': 52, 'ttl': 2592000},
                    { 'id': 2}),
                  ( ( 'test',
                      'test',
                      u'key1',
                      bytearray(b'\x1cJ\xce\xa7\xd4Vj\xef+\xdf@W\xa5\xd8o\x8d:\xc9\xf4\xde')),
                    { 'gen': 52, 'ttl': 2592000},
                    { 'a': 1, 'id': 1})]

        .. note:: As of client 7.0.0 and with server >= 6.0 results and the scan policy
         "partition_filter" see :ref:`aerospike_partition_objects` can be used to specify which partitions/records
         results will scan. See the example below.

         .. code-block:: python

            # This is an example of scanning partitions 1000 - 1003.
            import aerospike


            scan = client.scan("test", "demo")

            policy = {
                "partition_filter": {
                    "begin": 1000,
                    "count": 4
                },
            }

            # NOTE that these will only be non 0 if there are records in partitions 1000 - 1003
            # results will be the records in partitions 1000 - 1003
            results = scan.results(policy=policy)



    .. method:: foreach(callback[, policy[, options[, nodename]]])

        Invoke the *callback* function for each of the records streaming back \
        from the scan.

        :param callable callback: the function to invoke for each record.
        :param dict policy: optional :ref:`aerospike_scan_policies`.
        :param dict options: the :ref:`aerospike_scan_options` that will apply to the scan.
        :param str nodename: optional Node ID of node used to limit the scan to a single node.

        .. note::
            A :ref:`aerospike_record_tuple` is passed as the argument to the callback function.
            If the scan is using the "partition_filter" scan policy the callback will receive two arguments
            The first is a :class:`int` representing partition id, the second is the same :ref:`aerospike_record_tuple`
            as a normal callback.

        .. code-block:: python

            import aerospike
            import pprint

            pp = pprint.PrettyPrinter(indent=2)
            config = { 'hosts': [ ('127.0.0.1',3000)]}
            client = aerospike.client(config)

            client.put(('test','test','key1'), {'id':1,'a':1},
                policy={'key':aerospike.POLICY_KEY_SEND})
            client.put(('test','test','key2'), {'id':2,'b':2},
                policy={'key':aerospike.POLICY_KEY_SEND})

            def show_key(record):
                key, meta, bins = record
                print(key)

            scan = client.scan('test', 'test')
            scan_opts = {
              'concurrent': True,
              'nobins': True
            }
            scan.foreach(show_key, options=scan_opts)
            client.close()

        .. note::

            We expect to see:

            .. code-block:: python

                ('test', 'test', u'key2', bytearray(b'\xb2\x18\n\xd4\xce\xd8\xba:\x96s\xf5\x9ba\xf1j\xa7t\xeem\x01'))
                ('test', 'test', u'key1', bytearray(b'\x1cJ\xce\xa7\xd4Vj\xef+\xdf@W\xa5\xd8o\x8d:\xc9\xf4\xde'))

        .. note:: To stop the stream return ``False`` from the callback function.

            .. code-block:: python

                import aerospike

                config = { 'hosts': [ ('127.0.0.1',3000)]}
                client = aerospike.client(config)

                def limit(lim, result):
                    c = [0] # integers are immutable so a list (mutable) is used for the counter
                    def key_add(record):
                        key, metadata, bins = record
                        if c[0] < lim:
                            result.append(key)
                            c[0] = c[0] + 1
                        else:
                            return False
                    return key_add

                scan = client.scan('test','user')
                keys = []
                scan.foreach(limit(100, keys))
                print(len(keys)) # this will be 100 if the number of matching records > 100
                client.close()

        .. note:: As of client 7.0.0 and with server >= 6.0 foreach and the scan policy
         "partition_filter" see :ref:`aerospike_partition_objects` can be used to specify which partitions/records
         foreach will scan. See the example below.

         .. code-block:: python

            # This is an example of scanning partitions 1000 - 1003.
            import aerospike


            partitions = []

            def callback(part_id, input_tuple):
                print(part_id)
                partitions.append(part_id)

            scan = client.scan("test", "demo")

            policy = {
                "partition_filter": {
                    "begin": 1000,
                    "count": 4
                },
            }

            scan.foreach(callback, policy)


            # NOTE that these will only be non 0 if there are records in partitions 1000 - 1003
            # should be 4
            print(len(partitions))

            # should be [1000, 1001, 1002, 1003]
            print(partitions)

    .. method:: execute_background([, policy])

        Execute a record UDF on records found by the scan in the background. This method returns before the scan has completed.
        A UDF can be added to the scan with :meth:`Scan.apply`.

        :param dict policy: optional :ref:`aerospike_write_policies`.

        :return: a job ID that can be used with :meth:`~aerospike.Client.job_info` to track the status of the ``aerospike.JOB_SCAN``, as it runs in the background.

        .. note::
            Python client version 3.10.0 implemented scan execute_background.

            .. include:: examples/scan/top.py
                :code: python

            .. include:: examples/scan/my_udf.lua
                :code: lua

    .. method:: paginate()

        Makes a scan instance a paginated scan.
        Call this if you are using the "max_records" scan policy and you need to scan data in pages.

        .. note::
            Calling .paginate() on a scan instance causes it to save its partition state.
            This can be retrieved later using .get_partitions_status(). This can also be done using the
            partition_filter policy.

        .. code-block:: python

            # scan 3 pages of 1000 records each.

            import aerospike

            pages = 3
            page_size = 1000
            policy = {"max_records": 1000}

            scan = client.scan('test', 'demo')

            scan.paginate()

            # NOTE: The number of pages queried and records returned per page can differ
            # if record counts are small or unbalanced across nodes.
            for page in range(pages):
                records = scan.results(policy=policy)

                print("got page: " + str(page))

                if scan.is_done():
                    print("all done")
                    break

            # This id can be used to paginate queries.

    .. method:: is_done()

        If using scan pagination, did the previous paginated or partition_filter scan using this scan instance return all records?

        :return: A :class:`bool` signifying whether this paginated scan instance has returned all records.

        .. code-block:: python

            import aerospike

            policy = {"max_records": 1000}

            scan = client.scan('test', 'demo')

            scan.paginate()

            records = scan.results(policy=policy)

            if scan.is_done():
                print("all done")

            # This id can be used to monitor the progress of a paginated scan.

    .. method:: get_partitions_status()

        Get this scan instance's partition status. That is which partitions have been queried and which have not.
        The returned value is a :class:`dict` with partition id, :class:`int`, as keys and :class:`tuple` as values.
        If the scan instance is not tracking its partitions, the returned :class:`dict` will be empty.

        .. note::
            A scan instance must have had .paginate() called on it in order retrieve its
            partition status. If .paginate() was not called, the scan instance will not save partition status.

        :return: a :class:`tuple` of form (id: :class:`int`, init: class`bool`, done: class`bool`, digest: :class:`bytearray`).
            See :ref:`aerospike_partition_objects` for more information.

        .. code-block:: python

            # This is an example of resuming a scan using partition status.
            import aerospike


            for i in range(15):
                key = ("test", "demo", i)
                bins = {"id": i}
                client.put(key, bins)

            records = []
            resumed_records = []

            def callback(input_tuple):
                record, _, _ = input_tuple

                if len(records) == 5:
                    return False

                records.append(record)

            scan = client.scan("test", "demo")
            scan.paginate()

            scan.foreach(callback)

            # The first scan should stop after 5 records.
            assert len(records) == 5

            partition_status = scan.get_partitions_status()

            def resume_callback(part_id, input_tuple):
                record, _, _ = input_tuple
                resumed_records.append(record)

            scan_resume = client.scan("test", "demo")

            policy = {
                "partition_filter": {
                    "partition_status": partition_status
                },
            }

            scan_resume.foreach(resume_callback, policy)

            # should be 15
            total_records = len(records) + len(resumed_records)
            print(total_records)

            # cleanup
            for i in range(15):
                key = ("test", "demo", i)
                client.remove(key)

.. _aerospike_scan_policies:

Policies
========

.. object:: policy

    A :class:`dict` of optional scan policies which are applicable to :meth:`Scan.results` and :meth:`Scan.foreach`. See :ref:`aerospike_policies`.

    See :ref:`aerospike_base_policies` as well.

    .. hlist::
        :columns: 1

        * **durable_delete** :class:`bool`
            | Perform durable delete (requires Enterprise server version >= 3.10)
            | If the transaction results in a record deletion, leave a tombstone for the record.
            |
            | Default: ``False``
        * **records_per_second** :class:`int`
            | Limit the scan to process records at records_per_second.
            | Requires server version >= 4.7.0.
            |
            | Default: ``0`` (no limit).
        * **max_records** :class:`int`
            | Approximate number of records to return to client.
            | This number is divided by the number of nodes involved in the scan.
            | The actual number of records returned may be less than max_records if node record counts are small and unbalanced across nodes.
            |
            | Default: ``0`` (No Limit).

            .. note:: Requires Aerospike server version >= 6.0
        * **partition_filter** :class:`dict`
            | A dictionary of partition information used by the client
            | to perform partition scans. Useful for resuming terminated scans and
            | scanning particular partitions/records.
            |
            |   See :ref:`aerospike_partition_objects` for more information.
            |
            | Default: ``{}`` (All partitions will be scanned).

        * **replica**
            | One of the :ref:`POLICY_REPLICA` values such as :data:`aerospike.POLICY_REPLICA_MASTER`
            |
            | Default: ``aerospike.POLICY_REPLICA_SEQUENCE``
        * **ttl** (:class:`int`)
            The default time-to-live (expiration) of the record in seconds. This field will only be used on
            background scan writes if :py:attr:`aerospike.Scan.ttl` is set to
            :data:`aerospike.TTL_CLIENT_DEFAULT`.

            There are also special values that can be set for this field. See :ref:`TTL_CONSTANTS`.

.. _aerospike_scan_options:

Options
=======

.. object:: options

    A :class:`dict` of optional scan options which are applicable to :meth:`Scan.foreach`.

    .. hlist::
        :columns: 1

        * **nobins** :class:`bool`
            | Whether to return the *bins* portion of the :ref:`aerospike_record_tuple`.
            |
            | Default ``False``.
        * **concurrent** :class:`bool`
            | Whether to run the scan concurrently on all nodes of the cluster.
            |
            | Default ``False``.
        * **percent** :class:`int`
            | Deprecated in version 6.0.0, will be removed in a coming release.
            | No longer available with server 5.6+.
            | Use scan policy max_records instead.
            | Percentage of records to return from the scan.
            |
            | Default ``100``.

    .. versionadded:: 1.0.39