airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] yangaws commented on a change in pull request #4091: [AIRFLOW-2524] Update SageMaker hook, operator and sensor for training, tuning and transform
Date Wed, 31 Oct 2018 10:22:58 GMT
yangaws commented on a change in pull request #4091: [AIRFLOW-2524] Update SageMaker hook,
operator and sensor for training, tuning and transform
URL: https://github.com/apache/incubator-airflow/pull/4091#discussion_r229635402
 
 

 ##########
 File path: airflow/contrib/hooks/sagemaker_hook.py
 ##########
 @@ -16,299 +16,793 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
-import copy
+import tarfile
+import tempfile
 import time
+import os
+import collections
+import functools
+from datetime import datetime
+
+import botocore.config
 from botocore.exceptions import ClientError
 
 from airflow.exceptions import AirflowException
 from airflow.contrib.hooks.aws_hook import AwsHook
 from airflow.hooks.S3_hook import S3Hook
 
 
+class LogState(object):
+    STARTING = 1
+    WAIT_IN_PROGRESS = 2
+    TAILING = 3
+    JOB_COMPLETE = 4
+    COMPLETE = 5
+
+
+# Position is a tuple that includes the last read timestamp and the number of items that
were read
+# at that time. This is used to figure out which event to start with on the next read.
+Position = collections.namedtuple('Position', ['timestamp', 'skip'])
+
+
+def argmin(arr, f):
+    """Return the index, i, in arr that minimizes f(arr[i])"""
+    m = None
+    i = None
+    for idx, item in enumerate(arr):
+        if item is not None:
+            if m is None or f(item) < m:
+                m = f(item)
+                i = idx
+    return i
+
+
+def some(arr):
+    """Return True iff there is an element, a, of arr such that a is not None"""
+    return functools.reduce(lambda x, y: x or (y is not None), arr, False)
 
 Review comment:
   New knowledge got, thanks! Updated.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message