Follow

Importing from cron task

This article assumes a certain level of understanding of technical concepts.

Contents

  1. How it works
  2. Updating from a local file
  3. Updating from a URL
  4. The Cron Command
  5. Creating a recurring Cron Task using CPanel
  6. Create a recurring Cron Task using the command line
  7. Partial Imports with Cron

The Total Import PRO module can be run from the command line in an SSH session, or automatically on a Cron Task, provided your webhost allows you to use these functions.

This gives you an extra step of automation over running it manually from the admin of your store. It also allows you to avoid 30 second time limits usually applied by shared webhosts' default PHP settings.

How it works

When you go through Steps 1-5, the settings you choose are saved. If you run the module from the command line, either by SSH or a Cron Task, these settings will be used to import your products.

Total Import PRO includes a file called cron_index.php, which will be placed in the admin/ folder of your
store for OpenCart installations, or inside your modules/totalimportpro/ folder for PrestaShop. This is the entry point to the Total Import PRO for SSH or Cron Task. It will bypass the normal security system, however it has its own security so that it can only be run from the command line (not through the webserver). This will prevent unwanted access.

The CLI version only runs Step 1, 3 and 5 of the import.

There are two ways you may wish to update your store automatically: from a feed file on your filesystem that you or another application is uploading regularly, or from a feed file that can be fetched from a URL.

Updating from a local file

To update from a local file, you will need to upload the file to your store. The Total Import PRO module will search for it in admin/feed.txt for OpenCart, and modules/totalimportpro/feed.txt for PrestaShop. Another process can update this file, and when your Total Import PRO runs next it will pick up the latest feed.txt file.

Updating from a URL

To update from a URL, simply select the URL option on Step 1, and input the full URL where the feed can be found. If you cannot access the feed through your browser, it is unlikely that the Total Import PRO will be able to either. This is a good test to run in your browser before automating the system. The Total Import
PRO will search for

The Cron Command

Use the following command to run the script on a Cron Task or through SSH:

OpenCart

php /home/opencart/admin/cron_index.php

PrestaShop

php /home/prestashop/modules/totalimportpro/cron_index.php

You will need to replace /home/opencart/ with the full path to your installation. If you have multiple saved profiles and you want to run each of them on an automatic schedule, you can set up one cron task per profile to do this. Cron tasks should not be running concurrently as they share crucial resources.

To run a certain profile, modify the cron command to include your profile name at the end, like so (where you have saved your profile as “profile_name”):

OpenCart

php /home/opencart/admin/cron_index.php profile_name

PrestaShop

php /home/prestashop/modules/totalimportpro/cron_index.php profile_name

Important

If your profile name has a space in it, you will need to surround the name with speech marks. e.g profile name should be "profile name"

Creating a recurring Cron Task using CPanel

 To set up a recurring Cron Task for your import using Cpanel, you will need to specify how often you want your task to be called. Only one import can be running at a time, so you must allow enough time for your import to finish before scheduling a second import.

To set up a Cron Task which runs once a day at midnight you would use a command like this:

 

Create a recurring Cron Task using the command line

The exact command for setting this up will depend on your host environment, if you have any trouble then you should contact your web host and they will be able to help you set up a cron task.

To create a new cron task you need to edit your crontab file, to edit this run this command:

crontab -e

To add a new crontask which runs once a day at midnight, add a line to the file with this command:

php /home/opencart/admin/cron_index.php my_profile

The last step is to save your crontab file, and you're done!


Partial Imports with Cron

You can import a partial amount of products through cron by setting a range parameter. See the command below:

OpenCart

php /home/opencart/admin/cron_index.php profile_name 1-100

PrestaShop

php /home/prestashop/modules/totalimportpro/cron_index.php profile_name 1-100

This will import the first 100 items from the feed in the profile_name settings profile.

Was this article helpful?
4 out of 4 found this helpful
Have more questions? Submit a request

Comments

  • Avatar
    Aat

    Hi Justin,

    How do I rename the uploaded CSV to feed.txt  automatically?

    There is an update daily with a new name every day.

     

    If I run cron_index.php from my browser(ie9) it is also updating the store ,

    Easy but not very safe.

     

    Greetings Aat

  • Avatar
    Milan

    Hi Justin ,

    how can I find the full path to my OpenCart installation.

    Nice video by the way,it explains a lot

  • Avatar
    HostJars

    Aat - you can update the feed.txt to admin/feed.txt manually, or you can use the file system, url upload or ftp upload options.

    Milan - if you take a look in config.php file in your store there will be a line something like:

    define('DIR_STORE', '/home/store/www/');

    In this example /home/store/www is your store directory.

  • Avatar
    Milan

    Hi Justin,

    can you tell me why would my cron works fine with  defaul php  like " /home/opencart/admin/cron_index.php" ,but not when I use profil name "/home/opencart/admin/cron_index.php profile_name"?

    It says "sript doesnt exist" or "bad script"

  • Avatar
    Paul

    Our store has 36,000+ products and we'd like to update them 2 times daily. We have successfully uploaded our data feed with both your CSV Import PRO and Total Import PRO. We're having problems with updating them now that they are in the store.  

    We are getting this error on our cron task email notifications after they run: Set-Cookie: PHPSESSID=o76d4a2rkcrt9f5mpo0fk7gb77 path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Content-type: text/html

    The cron jobs have been set up in our cpanel and are scheduled for only 1 update per day until we figure out the problem. We've broken down our 36K product data feed into 2 files that are 5+MB and 3+MB. Then we are running our cron jobs as this (for example): _php /home/opencart/admin/cron_index.php profile_name 1-1500. _We've scheduled multiple (24) 1500 products imports at 30 min intervals so the cron jobs don't overlap.

    Questions: What is the problem?

    Do we need to empty our cache after every import using your module?

    Can I even run these cron tasks through my cpanel?

    We can run an import through the admin panel with up to 5000 products and get a successful import (except for a 500 error page... once we hit the back button in the browser we get the admin panel and a message of a successful import of 0 products and update of 5000 products). Is it common to get a 500 error page even when importing just 1000 products?  It seems to happen to us everytime we import using Total Import PRO.

    Cheers,

    Paul

  • Avatar
    Jammie

    Paul - did you resolve this issue, I'm getting the exact same error.

  • Avatar
    Przemyslaw Jan Krojcer

    I am getting almost the same error as above, except in my case it is exactly:

    Status: 500 Internal Server Error

    Set-Cookie: PHPSESSID=d48fc2vv3haovtd30oj9n2dpd5; path=/

    Expires: Thu, 19 Nov 1981 08:52:00 GMT

    Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0

    Pragma: no-cache

    Content-type: text/html

    Which is really awkward because it was working fine earlier today, now it's spitting out that dreaded Internal Error 500 which I can't troubleshoot. I checked the log, and it's empty too. I don't know what to do, I can't auto-update the feed and this is the main reason why I purchased Total Import PRO over XML Import Pro.

  • Avatar
    Przemyslaw Jan Krojcer

    Okay, to all the people that are getting weird errors trying to run the Cron task. Try specifying the php.ini file. I'm not sure where PHP looks for it when you run the task through either Cron or SSH, but in either case, using the following command:

    php -c ~/public_html/store/php.ini ~/public_html/store/admin/cron_index.php Profile_Name

    Where the -c switch signifies the use of a custom php.ini file, and following the switch is the path to the php.ini file. Above are my paths, make sure you update the above to reflect your setup. Anyway, the reason I was getting an Internal Error_ 500_ was because of the execution time being exceeded. Evidently, any errors with the Cron script won't show up in your regular OpenCart error.txt file (the one visible in System > Error Logs. _However, it should update the error_log_ file in your OpenCart admin directory, however you might have to make sure you have error reporting enabled.

  • Avatar
    cor segers

    Hello Justin,

     

    i started posting on the forum viewtopic.php?f=112&t=118957&p=473040#p473040 but it think it is better to put / continue it here.

    I tried to run a cronjob, when I check the server cronlog I can see it started, but when I look in my shop and server folders nothing has happened.

    2014-02-03 9:50:41 - PHP Notice:  Undefined index: settings_groupname in /home/websites/domains/myopencartshop.nl/public_html/admin/controller/tool/total_import.php on line 74

    2014-02-03 9:50:41 - PHP Notice:  Undefined index: settings_groupname in /home/websites/domains/myopencartshop.nl/public_html/admin/controller/tool/total_import.php on line 75

     

    These are the errors I see in opencart log.

    Tried these versions for cron: (settings daily, 13.00 hours)

    php /home/websites/domains/myopencartshop.nl/public_html/admin/cron_index.php toytrade  

    /usr/bin/wget http://www.myopencartshop.nl/admin/cron_index.php toytrade

    php -c ~/home/websites/domains/myopencartshop.nl/public_html/php.ini ~/home/websites/domains/myopencartshop.nl/public_html/admin/cron_index.php toytrade

     

    When I check the server cronlogs I see this:

    Feb  3 14:00:01 vps33 CROND[31178]: (websites) CMD (php /home/websites/domains/myopencartshop.nl/public_html/admin/cron_index.php toytrade)

     

    So tested 3 crons, can see them starting but nothing happens ??

     

    Can you help me ??

  • Avatar
    HostJars

    The first one looks best, you may wish to add /usr/bin/php instead of php, but it should work either way normally. I've seen your ticket so will help you out there.

  • Avatar
    HostJars

    Folks just be careful that you don't set up cron tasks so that imports may overlap. Imports require the same resources, so overlapping cron tasks are dangerous. There is actually code built into the cron_index.php file to prevent crons running concurrently. This code will prevent your cron import from running. If a cron fails and exits without completing, you will typically have to wait an hour before you can run another cron import, while the cron lock remains and until it expires.

  • Avatar
    Will

    Thank you for the detailed guide, and also thanks to Przemyslaw Jan Krojcer for his comment above which resolved the timeout issue we were having. We've successfully imported over 270,000 products using Cron Task with Total Import Pro =)

  • Avatar
    Juraj Filakovszky

    Hi, can somebody share with me the exact working php.ini content? I got this error message with cron update:

    Set-Cookie: PHPSESSID=d48f78zsj56td30oj9n2dpd5; path=/

    Expires: Thu, 19 Nov 1981 12:03:00 GMT

    Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0

    Pragma: no-cache

    Content-type: text/html

     

    and can't go further. Inside opencart everything works fine, but with cron does not. I have only 27 MB xml file, feed.txt is ready, updated on server, but database is not updated.

     

    Please help me. Thanks.

    George

  • Avatar
    Wouter van Domselaar

    Hello, can someone help me?

    Get the following errors during the cron jobs. The rights are all good. I use opencart 2.0.1.1:
    PHP Warning: fopen(/var/www/vhosts/desexshop.eu/httpdocs/system/logs/error.log): failed to open stream: Permission denied in /var/www/vhosts/desexshop.eu/httpdocs/system/library/log.php on line 6
    PHP Notice: unserialize(): Error at offset 2 of 128 bytes in /var/www/vhosts/desexshop.eu/httpdocs/admin/controller/tool/total_import.php on line 120
    PHP Warning: fopen(/var/www/vhosts/desexshop.eu/httpdocs/admin/feed.txt): failed to open stream: Permission denied in /var/www/vhosts/desexshop.eu/httpdocs/admin/model/tool/total_import.php on line 1114
    PHP Warning: curl_setopt(): supplied argument is not a valid File-Handle resource in /var/www/vhosts/desexshop.eu/httpdocs/admin/model/tool/total_import.php on line 1126
    PHP Warning: fclose() expects parameter 1 to be resource, boolean given in /var/www/vhosts/desexshop.eu/httpdocs/admin/model/tool/total_import.php on line 1134
    PHP Notice: unserialize(): Error at offset 2379 of 2584 bytes in /var/www/vhosts/desexshop.eu/httpdocs/admin/controller/tool/total_import.php on line 120
    PHP Notice: unserialize(): Error at offset 2 of 128 bytes in /var/www/vhosts/desexshop.eu/httpdocs/admin/controller/tool/total_import.php on line 120
    PHP Notice: unserialize(): Error at offset 2379 of 2584 bytes in /var/www/vhosts/desexshop.eu/httpdocs/admin/controller/tool/total_import.php on line 120
    Success: Added 0 products, Updated 13169 productsPHP Warning: fclose() expects parameter 1 to be resource, boolean given in /var/www/vhosts/desexshop.eu/httpdocs/system/library/log.php on line 14

  • Avatar
    Youp Schouten

    Is it possible to use a cron job that updates only the stock? If so, what should I put in? php /home/opencart/admin/cron_index.php profile_name.................. ?Thank you for helping.

  • Avatar
    Augusto Murri

    Hello, i think there is an error using Prestashop as MultiStore, in particolar the lines from 1575 to 1583 shoul be replace with:

        if (isset($settings['id_shop_list']))
        {
            $sql .= ' JOIN  `'._DB_PREFIX_.'product_shop` ps ON p.id_product = ps.id_product WHERE ps.id_shop IN (';
            $store_id_array = array();
            foreach ($settings['id_shop_list'] as $key => $store_id)
                $store_id_array[] = $store_id;
            $sql .= implode(', ', $store_id_array);
            $sql .= ')';
        }
    

    $settings['store'] doesn't exists, i think $settings['id_shop_list'] is correct

    let me know if i am wrong

    thanks

  • Avatar
    Augusto Murri

    Hello, another should be, when i choose RESET STORE it shold drop product of the specified store and not all products and combinations!

    thanks

  • Avatar
    Paul Starling

    In Opencart 3 what has happened to the cron_index.php file?

    Does it still exist?

Powered by Zendesk