hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergey Shelukhin <ser...@hortonworks.com>
Subject Re: file too large - itests/thirdparty/spark-1.2.0-bin-hadoop2-without-hive.tgz
Date Tue, 24 Nov 2015 19:37:49 GMT
Is there any update on this?

On 15/11/20, 14:36, "Owen O'Malley" <omalley@apache.org> wrote:

>I just sent off my message to board@apache with our ask. I'll keep you
>informed as to the outcome.
>
>.. Owen
>
>On Fri, Nov 20, 2015 at 2:07 PM, Owen O'Malley <omalley@apache.org> wrote:
>
>> Ok, I just talked to them over on the Infra hipchat and Sam Ruby
>>suggested
>> that I take it to the board to get permission to replace our master
>>branch.
>>
>> .. Owen
>>
>> On Fri, Nov 20, 2015 at 1:41 PM, Owen O'Malley <omalley@apache.org>
>>wrote:
>>
>>> I've been maintaining the master-fixed branch, but I missed the fact
>>>that
>>> infra asked us to try pushing it ourself. Let me try that.
>>>
>>> .. Owen
>>>
>>> On Fri, Nov 20, 2015 at 12:27 PM, Jimmy Xiang <jxiang@cloudera.com>
>>> wrote:
>>>
>>>> I can get the latest of master branch, clean it up, and force-push to
>>>> master-fixed again.
>>>>
>>>> Another solution is to abandon branch master, then use a new branch
>>>>like
>>>> "trunk".
>>>>
>>>> Thanks,
>>>> Jimmy
>>>>
>>>> On Fri, Nov 20, 2015 at 12:07 PM, Sergey Shelukhin <
>>>> sergey@hortonworks.com>
>>>> wrote:
>>>>
>>>> > Did this ever go anywhere?
>>>> > Infra told us to try ourselves.
>>>> > I wonder if we should do it before releasing 2.0...
>>>> >
>>>> > I am assuming master-fixed diverged by now. It can be rebased as
>>>> follows:
>>>> >
>>>> >  git rebase --onto master-fixed `git log master --grep="$(git show
>>>>-s
>>>> > master-fixed --format="%s")" --format="%H"` master
>>>> >
>>>> >  git push origin master:master-fixed (!!!)
>>>> >
>>>> > Or with safety checks:
>>>> > * OLD_HEAD=`git log master --grep="$(git show -s master-fixed
>>>> > --format="%s")" --format="%H"`
>>>> > * Optional git show $OLD_HEAD and git show master-fixed to see if
>>>>they
>>>> > match
>>>> > * git rebase --onto master-fixed `git log master --grep="$(git show
>>>>-s
>>>> > master-fixed --format="%s")" --format="%H"` master
>>>> >   This will update the branch correctly but change master, not
>>>> > master-fixed (because essentially you are rebasing new commits
>>>>added to
>>>> > master on top of master-fixed)… If someone knows a better way to do
>>>>it
>>>> let
>>>> > me know.
>>>> > * Optionally git diff master origin/master to check.
>>>> > * git push origin master:master-fixed.
>>>> > * For good measure, git reset --hard origin/master.
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On 15/11/6, 13:36, "Owen O'Malley" <omalley@apache.org> wrote:
>>>> >
>>>> > >Ok, I've filled the infra jira for replacing the master branch:
>>>> > >
>>>> > > https://issues.apache.org/jira/browse/INFRA-10731
>>>> > >
>>>> > >.. Owen
>>>> > >
>>>> > >On Fri, Nov 6, 2015 at 9:36 AM, Jimmy Xiang <jxiang@cloudera.com>
>>>> wrote:
>>>> > >
>>>> > >> Done. Branch master-fixed is clean now. What should we do next?
>>>> > >>
>>>> > >> On Fri, Nov 6, 2015 at 9:26 AM, Jimmy Xiang <jxiang@cloudera.com>
>>>> > wrote:
>>>> > >>
>>>> > >> > Let me do that.
>>>> > >> >
>>>> > >> > On Fri, Nov 6, 2015 at 9:24 AM, Owen O'Malley
>>>><omalley@apache.org
>>>> >
>>>> > >> wrote:
>>>> > >> >
>>>> > >> >> Can someone push a fixed master history to "master-fixed"?
>>>>Then
>>>> we
>>>> > >>can
>>>> > >> >> start a discussion with infra about replacing master
with
>>>> > >>master-fixed.
>>>> > >> >>
>>>> > >> >> .. Owen
>>>> > >> >>
>>>> > >> >> On Fri, Nov 6, 2015 at 9:04 AM, Jimmy Xiang
>>>><jxiang@cloudera.com
>>>> >
>>>> > >> wrote:
>>>> > >> >>
>>>> > >> >> > Right. The file is not in source control any
more. But it
>>>>was
>>>> > >>there at
>>>> > >> >> some
>>>> > >> >> > moment. That's why it is in the git history.
So we need to
>>>> clean up
>>>> > >> the
>>>> > >> >> git
>>>> > >> >> > history.
>>>> > >> >> >
>>>> > >> >> > To push to a branch other than master, for example,
>>>> hive-11890, as
>>>> > >>a
>>>> > >> >> > work-around, you can try
>>>> > https://rtyley.github.io/bfg-repo-cleaner/
>>>> > >> to
>>>> > >> >> do
>>>> > >> >> > the clean up first, then push.
>>>> > >> >> >
>>>> > >> >> > On Fri, Nov 6, 2015 at 9:00 AM, Jimmy Xiang <
>>>> jxiang@cloudera.com>
>>>> > >> >> wrote:
>>>> > >> >> >
>>>> > >> >> > > I ran into the same problem too. What we
need to do is to
>>>> clean
>>>> > >>up
>>>> > >> the
>>>> > >> >> > git
>>>> > >> >> > > history. However, the master branch can't
be
>>>>force-pushed. I
>>>> was
>>>> > >> >> > wondering
>>>> > >> >> > > if this is something the infra team can
help.
>>>> > >> >> > >
>>>> > >> >> > > On Fri, Nov 6, 2015 at 8:54 AM, Xuefu Zhang
<
>>>> xzhang@cloudera.com
>>>> > >
>>>> > >> >> wrote:
>>>> > >> >> > >
>>>> > >> >> > >> I don't quite follow. That file is not
in source control,
>>>> but
>>>> > >>only
>>>> > >> >> > >> downloaded when running test. Why a
git push will mass
>>>>with
>>>> that
>>>> > >> >> file?
>>>> > >> >> > >>
>>>> > >> >> > >>
>>>> > >> >> > >> On Fri, Nov 6, 2015 at 7:44 AM, Owen
O'Malley
>>>> > >><omalley@apache.org>
>>>> > >> >> > wrote:
>>>> > >> >> > >>
>>>> > >> >> > >> > Github apparently has a limit on
pushes to personal
>>>> accounts
>>>> > >>that
>>>> > >> >> no
>>>> > >> >> > >> file
>>>> > >> >> > >> > may be larger than 100mb. When
I try to push a branch
>>>>to
>>>> my
>>>> > >> >> personal
>>>> > >> >> > >> clone
>>>> > >> >> > >> > of hive, I get:
>>>> > >> >> > >> >
>>>> > >> >> > >> > rockfleet:hive owen$ git push omalley
master:hive-11890
>>>> > >> >> > >> > > Counting objects: 21159, done.
>>>> > >> >> > >> > > Delta compression using up
to 4 threads.
>>>> > >> >> > >> > > Compressing objects: 100%
(6561/6561), done.
>>>> > >> >> > >> > > Writing objects: 100% (21159/21159),
110.16 MiB |
>>>>3.19
>>>> > >>MiB/s,
>>>> > >> >> done.
>>>> > >> >> > >> > > Total 21159 (delta 11142),
reused 18555 (delta 9014)
>>>> > >> >> > >> > > remote: error: GH001: Large
files detected. You may
>>>> want to
>>>> > >>try
>>>> > >> >> Git
>>>> > >> >> > >> Large
>>>> > >> >> > >> > > File Storage - https://git-lfs.github.com.
>>>> > >> >> > >> > > remote: error: Trace:
>>>>5cfe5408835563c7f661c3086c84930a
>>>> > >> >> > >> > > remote: error: See http://git.io/iEPt8g
for more
>>>> > >>information.
>>>> > >> >> > >> > > remote: error: File
>>>> > >> >> > >> > >
>>>> itests/thirdparty/spark-1.2.0-bin-hadoop2-without-hive.tgz
>>>> > >>is
>>>> > >> >> 103.62
>>>> > >> >> > >> MB;
>>>> > >> >> > >> > > this exceeds GitHub's file
size limit of 100.00 MB
>>>> > >> >> > >> > > To git@github.com:omalley/hive.git
>>>> > >> >> > >> > >  ! [remote rejected] master
-> hive-11890
>>>>(pre-receive
>>>> hook
>>>> > >> >> > declined)
>>>> > >> >> > >> > > error: failed to push some
refs to 'git@github.com:
>>>> > >> >> > omalley/hive.git'
>>>> > >> >> > >> >
>>>> > >> >> > >> >
>>>> > >> >> > >> > This makes it very difficult to
work with. Can we do
>>>> something
>>>> > >> else
>>>> > >> >> > with
>>>> > >> >> > >> > the file? If not, can we store
it with the git large
>>>>file
>>>> > >>storage
>>>> > >> >> as
>>>> > >> >> > >> > suggested in the error message?
>>>> > >> >> > >> >
>>>> > >> >> > >> > .. Owen
>>>> > >> >> > >> >
>>>> > >> >> > >>
>>>> > >> >> > >
>>>> > >> >> > >
>>>> > >> >> >
>>>> > >> >>
>>>> > >> >
>>>> > >> >
>>>> > >>
>>>> >
>>>> >
>>>>
>>>
>>>
>>

Mime
View raw message