Slow indexing process

  • Resolved flaviopv

    (@flaviusv)


    7 months, 3 weeks ago

    Hi,

    I would like to ask you a question, if you could help me.

    On the 1st day we started the indexing process, in 10 hours it has been sent to Typesense 99.965 documents.
    On the 2nd day of the indexing process, in 10 hours it has been sent to Typesense 336 documents.
    Today, on 3rd day of the indexing process, in 4 hours it has been sent to Typesense 166 documents.

    In total we have 310.000 posts.

    Do you have any idea why the indexing process became so slow?

    many thanks,

Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Contributor digamberpradhan

    (@digamberpradhan)

    7 months, 3 weeks ago

    Hi @flaviusv

    For large indexes it’s really very difficult to stop the bottleneck of database queries.
    To counteract this problem i would suggest that you export data / post type to a .txt file.
    Create a .sh file lets say read_file.sh and run this 

    # Check if a filename is provided
    if [ $# -eq 0 ]; then
        echo "Usage: $0 ids.txt"
        exit 1
    fi

    # Read the file line by line and echo each line
    while IFS= read -r line; do
        wp typesense index post --ids=$line
    done < "$1"

    Then run the .sh file: ./read_file.sh ids.txt
    This will be much more performant.

    Thread Starter flaviopv

    (@flaviusv)

    7 months, 3 weeks ago

    Hi,

    Thank you very much, your solution is working. Now we are able to send in avg. 10k posts/hr.

Viewing 2 replies - 1 through 2 (of 2 total)

The topic ‘Slow indexing process’ is closed to new replies.