Source code:Lib/asyncio/subprocess.py,Lib/asyncio/base_subprocess.py
This section describes high-level async/await asyncio APIs tocreate and manage subprocesses.
Python’s subprocess module disables SIGPIPE by default (SIGPIPE is sets to ignore). Most unix programs expect to run with SIGPIPE enabled. When running a single external program (e.g. With shell=False) this is usually not a problem. It does become a problem when running shell-pipes, or when the executed program runs sub-programs on its own. I've been trying to understand for a while now what's the difference between subprocess.call and subprocess.run. I know the last one is new on Python 3.5 and both are based on subprocess.Popen, but I'm not able to understand the difference yet.
Here’s an example of how asyncio can run a shell command andobtain its result:
will print:
Because all asyncio subprocess functions are asynchronous and asyncioprovides many tools to work with such functions, it is easy to executeand monitor multiple subprocesses in parallel. It is indeed trivialto modify the above example to run several commands simultaneously:
See also the Examples subsection.
Creating Subprocesses¶
asyncio.
create_subprocess_exec
(program, *args, stdin=None, stdout=None, stderr=None, loop=None, limit=None, **kwds)¶Create a subprocess.
The limit argument sets the buffer limit for
StreamReader
wrappers for Process.stdout
and Process.stderr
(if subprocess.PIPE
is passed to stdout and stderr arguments).Return a
Process
instance.See the documentation of
loop.subprocess_exec()
for otherparameters.Deprecated since version 3.8, will be removed in version 3.10: The loop parameter.
asyncio.
create_subprocess_shell
(cmd, stdin=None, stdout=None, stderr=None, loop=None, limit=None, **kwds)¶Run the cmd shell command.
The limit argument sets the buffer limit for
StreamReader
wrappers for Process.stdout
and Process.stderr
(if subprocess.PIPE
is passed to stdout and stderr arguments).Return a
Process
instance.See the documentation of
loop.subprocess_shell()
for otherparameters.Important
It is the application’s responsibility to ensure that all whitespace andspecial characters are quoted appropriately to avoid shell injectionvulnerabilities. The
shlex.quote()
function can be used to properlyescape whitespace and special shell characters in strings that are goingto be used to construct shell commands.Deprecated since version 3.8, will be removed in version 3.10: The loop parameter.
Note
The default asyncio event loop implementation on Windows does notsupport subprocesses. Subprocesses are available for Windows if a
ProactorEventLoop
is used.See Subprocess Support on Windowsfor details.See also
asyncio also has the following low-level APIs to work with subprocesses:
loop.subprocess_exec()
, loop.subprocess_shell()
,loop.connect_read_pipe()
, loop.connect_write_pipe()
,as well as the Subprocess Transportsand Subprocess Protocols.Constants¶
asyncio.subprocess.
PIPE
¶Can be passed to the stdin, stdout or stderr parameters.
If PIPE is passed to stdin argument, the
Process.stdin
attributewill point to a StreamWriter
instance.If PIPE is passed to stdout or stderr arguments, the
Process.stdout
andProcess.stderr
attributes will point to StreamReader
instances.asyncio.subprocess.
STDOUT
¶Special value that can be used as the stderr argument and indicatesthat standard error should be redirected into standard output.
asyncio.subprocess.
DEVNULL
¶Special value that can be used as the stdin, stdout or stderr argumentto process creation functions. It indicates that the special file
os.devnull
will be used for the corresponding subprocess stream.Interacting with Subprocesses¶
Both
create_subprocess_exec()
and create_subprocess_shell()
functions return instances of the Process class. Process is a high-levelwrapper that allows communicating with subprocesses and watching fortheir completion.asyncio.subprocess.
Process
¶An object that wraps OS processes created by the
create_subprocess_exec()
and create_subprocess_shell()
functions.This class is designed to have a similar API to the
subprocess.Popen
class, but there are somenotable differences:- unlike Popen, Process instances do not have an equivalent tothe
poll()
method; - the
communicate()
andwait()
methods don’t have atimeout parameter: use thewait_for()
function; - the
Process.wait()
methodis asynchronous, whereassubprocess.Popen.wait()
methodis implemented as a blocking busy loop; - the universal_newlines parameter is not supported.
This class is not thread safe.
See also the Subprocess and Threadssection.
wait
()¶Wait for the child process to terminate.
Set and return the
returncode
attribute.Note
This method can deadlock when using
stdout=PIPE
orstderr=PIPE
and the child process generates so much outputthat it blocks waiting for the OS pipe buffer to acceptmore data. Use the communicate()
method when using pipesto avoid this condition.communicate
(input=None)¶Interact with process:
- send data to stdin (if input is not
None
); - read data from stdout and stderr, until EOF is reached;
- wait for process to terminate.
The optional input argument is the data (
bytes
object)that will be sent to the child process.Return a tuple
(stdout_data,stderr_data)
.If either
BrokenPipeError
or ConnectionResetError
exception is raised when writing input into stdin, theexception is ignored. This condition occurs when the processexits before all data are written into stdin.If it is desired to send data to the process’ stdin,the process needs to be created with
stdin=PIPE
. Similarly,to get anything other than None
in the result tuple, theprocess has to be created with stdout=PIPE
and/orstderr=PIPE
arguments.Note, that the data read is buffered in memory, so do not usethis method if the data size is large or unlimited.
send_signal
(signal)¶Sends the signal signal to the child process.
Note
On Windows,
SIGTERM
is an alias for terminate()
.CTRL_C_EVENT
and CTRL_BREAK_EVENT
can be sent to processesstarted with a creationflags parameter which includesCREATE_NEW_PROCESS_GROUP
.terminate
()¶Stop the child process.
On POSIX systems this method sends
signal.SIGTERM
to thechild process.On Windows the Win32 API function
TerminateProcess()
iscalled to stop the child process.kill
()¶Kill the child.
On POSIX systems this method sends
SIGKILL
to the childprocess.On Windows this method is an alias for
terminate()
.stdin
¶Standard input stream (
StreamWriter
) or None
if the process was created with stdin=None
.stdout
¶Standard output stream (
StreamReader
) or None
if the process was created with stdout=None
.stderr
¶Standard error stream (
StreamReader
) or None
if the process was created with stderr=None
.Warning
Use the
communicate()
method rather thanprocess.stdin.write()
,awaitprocess.stdout.read()
orawaitprocess.stderr.read
.This avoids deadlocks due to streams pausing reading or writingand blocking the child process.pid
¶Process identification number (PID).
Note that for processes created by the
create_subprocess_shell()
function, this attribute is the PID of the spawned shell.returncode
¶Return code of the process when it exits.
A
None
value indicates that the process has not terminated yet.A negative value
-N
indicates that the child was terminatedby signal N
(POSIX only).Subprocess and Threads¶
Standard asyncio event loop supports running subprocesses from different threads bydefault.
On Windows subprocesses are provided by
ProactorEventLoop
only (default),SelectorEventLoop
has no subprocess support.On UNIX child watchers are used for subprocess finish waiting, seeProcess Watchers for more info.
Changed in version 3.8: UNIX switched to use
ThreadedChildWatcher
for spawning subprocesses fromdifferent threads without any limitation.Spawning a subprocess with inactive current child watcher raises
RuntimeError
.Note that alternative event loop implementations might have own limitations;please refer to their documentation.
See also
The Concurrency and multithreading in asyncio section.
Examples¶
An example using the
Process
class tocontrol a subprocess and the StreamReader
class to read fromits standard output.The subprocess is created by the
create_subprocess_exec()
function:See also the same examplewritten using low-level APIs.