celery==4.1.0
Django==1.9.7
requests==2.10.0
requests-futures==0.9.7
Реализую задачу загрузки видеофайлов в ВК, для асинхронных запросов использую requests-futures в джанго.
Как собираю задачи:
tasks = []
response = {}
for group in communities:
community = Communities.objects.get(id=group)
if community.social_network == 'vk':
post = Post.objects.get(community=community, message_id=message_id)
tasks.append(vk_wall_post.s(user_id=user.id, message_id=message.id,
community_id=community.id, post_id=post.id))
# vk_resp = vk_wall_post(user.token, message, community, post)
response.update({group: 'VK post in process...'})
# Celery callback
@task(name='CallbackNotifier', base=NotifierTask)
def callback_notifier():
return True
# Celery chord
callback = callback_notifier.s()
chord(tasks)(callback)
Загрузка файла на сервер вк, есть запросы инициализации и постинга, они тоже работают через requests-futures
session = FuturesSession()
with open(video_file_path, 'rb') as f:
form = encoder.MultipartEncoder({
'video_file': (video.name, f, 'application/octet-stream')
})
headers = {
'Prefer': 'respond-async',
'Content-Type': form.content_type
}
resp = session.post(video_upload_url, headers=headers, data=form)
response = resp.result()
if response.status_code != 200:
return "Request Entity Too Large"
else:
vk_video_id = response.json().get('video_id')
return vk_video_id
лог воркера селери
celery_1 | [2018-01-03 08:19:42,460: INFO/MainProcess] celery@d0d50ca50544 ready.
celery_1 | [2018-01-03 08:19:46,953: INFO/MainProcess] Events of group {task} enabled by remote.
celery_1 | [2018-01-03 08:20:48,560: INFO/MainProcess] Received task: api.controllers.vk_api.vk_wall_post[11659f79-d19c-4e8e-94ca-946d12dfa502]
celery_1 | [2018-01-03 08:20:48,567: INFO/MainProcess] Received task: api.controllers.vk_api.vk_wall_post[73947637-1567-4262-b5f1-174c9f7e8abe]
celery_1 | [2018-01-03 08:20:48,578: INFO/MainProcess] Received task: api.controllers.vk_api.vk_wall_post[8655511e-e87d-4802-93f8-025ad85533d5]
celery_1 | [2018-01-03 08:20:48,582: INFO/MainProcess] Received task: api.controllers.vk_api.vk_wall_post[167e2314-a0c2-4a25-be2a-aec4372d2fda]
celery_1 | [2018-01-03 08:20:48,756: INFO/ForkPoolWorker-1] Starting new HTTPS connection (1): api.vk.com
celery_1 | [2018-01-03 08:20:49,190: INFO/ForkPoolWorker-1] Starting new HTTPS connection (1): cs838419.userapi.com
celery_1 | [2018-01-03 08:21:36,080: INFO/ForkPoolWorker-1] Starting new HTTPS connection (1): api.vk.com
celery_1 | [2018-01-03 08:21:36,239: INFO/ForkPoolWorker-1] Task api.controllers.vk_api.vk_wall_post[11659f79-d19c-4e8e-94ca-946d12dfa502] succeeded in 47.67057052603923s: {'response': {'post_id': 950}}
celery_1 | [2018-01-03 08:21:36,239: WARNING/ForkPoolWorker-1] SUCCESS
celery_1 | [2018-01-03 08:21:36,240: WARNING/ForkPoolWorker-1] {'response': {'post_id': 950}}
celery_1 | [2018-01-03 08:21:36,240: WARNING/ForkPoolWorker-1] 11659f79-d19c-4e8e-94ca-946d12dfa502
celery_1 | [2018-01-03 08:21:36,240: WARNING/ForkPoolWorker-1] []
celery_1 | [2018-01-03 08:21:36,241: WARNING/ForkPoolWorker-1] {'message_id': 80, 'user_id': 4, 'post_id': 109, 'community_id': 1}
celery_1 | [2018-01-03 08:21:36,241: WARNING/ForkPoolWorker-1] None
celery_1 | [2018-01-03 08:21:36,241: WARNING/ForkPoolWorker-1] Task done
celery_1 | [2018-01-03 08:21:36,260: INFO/ForkPoolWorker-1] Starting new HTTPS connection (1): api.vk.com
celery_1 | [2018-01-03 08:21:36,374: INFO/ForkPoolWorker-1] Starting new HTTPS connection (1): cs838120.userapi.com
celery_1 | [2018-01-03 08:22:23,205: INFO/ForkPoolWorker-1] Starting new HTTPS connection (1): api.vk.com
celery_1 | [2018-01-03 08:22:23,457: INFO/ForkPoolWorker-1] Task api.controllers.vk_api.vk_wall_post[73947637-1567-4262-b5f1-174c9f7e8abe] succeeded in 47.20932572206948s: {'response': {'post_id': 41}}
celery_1 | [2018-01-03 08:22:23,458: WARNING/ForkPoolWorker-1] SUCCESS
celery_1 | [2018-01-03 08:22:23,458: WARNING/ForkPoolWorker-1] {'response': {'post_id': 41}}
celery_1 | [2018-01-03 08:22:23,458: WARNING/ForkPoolWorker-1] 73947637-1567-4262-b5f1-174c9f7e8abe
celery_1 | [2018-01-03 08:22:23,459: WARNING/ForkPoolWorker-1] []
celery_1 | [2018-01-03 08:22:23,459: WARNING/ForkPoolWorker-1] {'message_id': 80, 'user_id': 4, 'post_id': 112, 'community_id': 2}
celery_1 | [2018-01-03 08:22:23,459: WARNING/ForkPoolWorker-1] None
celery_1 | [2018-01-03 08:22:23,460: WARNING/ForkPoolWorker-1] Task done
celery_1 | [2018-01-03 08:22:23,476: INFO/ForkPoolWorker-1] Starting new HTTPS connection (1): api.vk.com
celery_1 | [2018-01-03 08:22:23,596: INFO/ForkPoolWorker-1] Starting new HTTPS connection (1): cs838212.userapi.com
celery_1 | [2018-01-03 08:23:09,542: INFO/ForkPoolWorker-1] Starting new HTTPS connection (1): api.vk.com
celery_1 | [2018-01-03 08:23:09,823: INFO/ForkPoolWorker-1] Task api.controllers.vk_api.vk_wall_post[8655511e-e87d-4802-93f8-025ad85533d5] succeeded in 46.3576935910387s: {'response': {'post_id': 15}}
celery_1 | [2018-01-03 08:23:09,824: WARNING/ForkPoolWorker-1] SUCCESS
celery_1 | [2018-01-03 08:23:09,824: WARNING/ForkPoolWorker-1] {'response': {'post_id': 15}}
celery_1 | [2018-01-03 08:23:09,824: WARNING/ForkPoolWorker-1] 8655511e-e87d-4802-93f8-025ad85533d5
celery_1 | [2018-01-03 08:23:09,824: WARNING/ForkPoolWorker-1] []
celery_1 | [2018-01-03 08:23:09,825: WARNING/ForkPoolWorker-1] {'message_id': 80, 'user_id': 4, 'post_id': 111, 'community_id': 3}
celery_1 | [2018-01-03 08:23:09,825: WARNING/ForkPoolWorker-1] None
celery_1 | [2018-01-03 08:23:09,825: WARNING/ForkPoolWorker-1] Task done
celery_1 | [2018-01-03 08:23:09,875: INFO/ForkPoolWorker-1] Starting new HTTPS connection (1): api.vk.com
celery_1 | [2018-01-03 08:23:10,002: INFO/ForkPoolWorker-1] Starting new HTTPS connection (1): cs514511.userapi.com
celery_1 | [2018-01-03 08:23:57,163: INFO/ForkPoolWorker-1] Starting new HTTPS connection (1): api.vk.com
celery_1 | [2018-01-03 08:23:57,406: INFO/ForkPoolWorker-1] Task api.controllers.vk_api.vk_wall_post[167e2314-a0c2-4a25-be2a-aec4372d2fda] succeeded in 47.57519669702742s: {'response': {'post_id': 30}}
Воркер выполняет запросы синхронно, как добиться параллельного выполнения?
Почему происходит блокировка, если использую не блокирующий req-futures?
